Tech TLDR;

Group Log Data by Timestamp in Python with Pandas

by

I often find myself with logs in the following format, exported in CSV. First column is timestamp in milliseconds, second column is timing for some request, also in milliseconds. Often I’d like to see the min, max and average request time, group by day and hour. Here is how to get this insight using Pandas […]

How to use Virtualenv in Python to Install Packages Locally

by

There are two philosophies when it comes to package installation, global first and local first. Global meaning all applications that rely on a certain package have access to the same copy of the library that was installed once. Local means that each project has its own folder of dependencies installed specifically for this project and […]

Crunching Honeypot IP Data with Pandas and Python

by

I am taking a cyber security class. This week’s assignment had us work on Honeypots. Honeypot is a server that pretends to have a vulnerability of sorts (open ports, old software etc.) and instead collects data on people who are trying to hack it. At the end of the experiment I ended up with some […]

How to Restart Python Script after Exception and Run it Forever

by

Here is a simple trick that I used to restart my python script after unhandled exception. Let’s say I have this simple script called test.py that I want to run forever. It will just wait 2 seconds and throw an error. I use the following script called forever in the same directory: It uses python […]

Transcribing Speech to Text with Python and Google Cloud Speech API

by

This tutorial will walk through using Google Cloud Speech API to transcribe a large audio file. All code and sample files can be found in speech-to-text GitHub repo. Transcribe large audio files using Python & our Cloud Speech API. @akras14 shows how https://t.co/dY56lmE0TD — Google Cloud (@googlecloud) January 11, 2018 View Post Sample Results This […]

How to read CSV file in Python

by

TLDR; Using pandas (pip install pandas) Using native csv.DictReader Reading as a List Let’s say we have the following CSV file, named actors.csv. You can download this file here. The first method demonstrated in Python docs would read the file as follows: Open the file Create a CSV reader Skip first line (header) For every line (row) in […]

Convert JSON to dot notation with Python

by

There are many use cases for converting JSON to a dot notation. For example, my current company used Hue to query our log data. Our logs are stored in JSON, and Hue queries expect a dot notation. For example, this JSON file Will look like this, when converted to dot notation: Note, I am starting […]

Parsing “Ask HN: Who is hiring?” with Python and Hacker News API

by

Have you heard of Hacker News? It’s a great mini social network dedicated to all things tech. Once a month they post a thread called “Ask HN: Who is hiring?”, where anyone can list their job openings. With hundreds of comments it quickly gets overwhelming. Turns out it’s very easy to get the same data […]

Simple Python Threading Example

by

At my work we have a Python library that interfaces to all our API micro services (that are written in Java/Scala). It is a very useful tool for debugging and working with our platform, so I spend a lot of my time in a Python REPL. Often times I find myself needing to hit multiple […]