Intro

Hi!

This article if suitable for Data Analysts or early stage developers, I'm sure there are multiple other ways of doing this and maybe even more efficiently.

This is meant to help you guys saving time coding and debugging. Here's my 50 cents.

Why do I use a log file?

Based on my experience, scripting and reviewing errors on console can help, at an early stage, as tracking the values of the variables, tables, lists, tuples..., and using Spyder is also very handy, to be sure that I'm getting the expected results.

However, ending up with thousands of lines of code, a variety of files and functions, and manipulating huge datasets, it gets very tricky tracking the progress of the scripts and especially identifying errors that might occur, and what an headache if the errors are not syntax errors that show up on the console!

Have you been there before? Share below :)

I'd say that I spend 80% of my coding time debugging and testing. I can't just trust my results without doing so, as a Data Analyst, to generating useless or contaminated results to then create my final reports.

How do I track my results using my log file?

On my code, everytime I'm generating a new result, after a data manipulation, or calculation, for example, I can simply print on file what I'm doing and print also a snippet of that result or check a var type. Very important as well, I catch all the errors using try-except, so I can track easily where did my code broke up and why.

Doind so, it helped me saving a lot of time searching for needles in a spaghetti soup.

Moreover, it also helps me building and keep tracking of my code structure and its O-value, and efficiency, by also checking the time stamps of each new step I'm tracking with the log file.

If something takes significantly more time, I jump back to my code and try to figure out a more efficient way of running it, the Pythonic way preferably.

How to create a log file?

You can use this to log all the steps that your app is doing, and keep track of changes and progress.

Super helpful for debugging!

#INIT THE LOG FILE in the __main__.py file and import the py file where you keep your functions, mine is called fun.py

import fun as fun

log_file_name = str()
log_file_name = fun.create_log(log_file_name)

# Note that I'm using my log_file_name as local to my main py file.

# In the fun.py file:
def create_log(file_name):  
    # Creates log_file.txt

    # define where to save the log file
    #homeDir = os.path.expanduser("~")

    # or define it manually:
    homeDir = r'C:\Users\Myname'
    # Alternatively, you can add a chose folder dialog box for the user
    # to set manually in, for example, a settings menu.

    # set the log file name, as you wish, here I use the simple standard log.log
    # again, you can specify this in a settings menu manually and let the user choose the log file name.

    file_name = homeDir + '\\log.log'

    # Open the file with writable permissions and create the first line

    f = open(file_name, "w")
    f.write('Created by @Francisco Rua')
    f.close()

    # file is now created, not empty and ready to take on more lines.

    return file_name

Every time I want to add a step log into my log file, I first define the message, the one I'll change every time, so I can use copy paste then print it on the console if needed and/or save it into the log file:

# Defining the new function in fun.py to add (append) a new line in the log file:

def add_log(file_name, add_str):
    # this append a new line in the logfile

    # get the time stamp
    now_time = datetime.now().strftime("%d/%m/%Y %H:%M:%S")
    # open and append to file
    f = open(file_name, "a")
    f.write('\n'+str(now_time) + ': ' + add_str)
    f.close()

Note:

I chose to save the filename in my __main__.py file, so I import __main__ as m in a new script file, alternatively we can make it a global variable, as you wish.

Anywhere I want in the project I use this :

import __main__ as m
import fun as fun

message = f'> example text : {_example_var_to_save}'

print(message) #if you wish

fun.add_log(m.log_file_name, message)

Consider using it where it needs and not everywhere, due to %timeit.

Every time you open, read/write, close a file, it is resource consuming. What you can do is have it on, extensively, while you're building the app. Then remove the unnecessary steps if debuug is no longer needed. Or you can also add a boolean variable called debugg, and these lines of code would only be printed/logged if the debugg == True.

In such case, use this for example:

if debugg == True : fun.add_log(m.log_file_name, message)

Final note

As a Data Analyst, you might be using sensitive information and because that file is stored locally, I'd recommend not to throw in it any sensitive information into the log file as for example, personal information, account information, explicitly any information that can lead to identifying a person, business or any confidential details.

I hope that this can help someone, and especially help you save a lot of time.

Data Analyst Journey