Monitoring   Instrumentation  

Instrumentation: system calls: an amazing interface for instrumentation

By Guest Blogger  |   Last modified on December 1, 2018

When you’re debugging, there are two basic ways you can poke at something. You can:

  • create new instrumentation (like “adding print statements”)
  • use existing instrumentation (“look at print statements you already added”, “use Wireshark”)

When your program is already running and already doing some TERRIBLE THING YOU DO NOT WANT, it is very nice to be able to ask questions of it (“dear sir, what ARE you doing”) without having to recompile it or restart or anything.

I think about asking questions of a program in terms of “what interfaces does it have that I can observe?”. Can I tell which system calls that program is using? Can I look at its network requests? Can I easily see database queries?

interfaces

In this post, we’ll talk about my very favourite interface: system calls

what’s a system call?

Your program does not know how to open files: it doesn’t know what a hard drive is, or how to read a filesystem, or any of that. However, your operating system does know all that stuff!

System calls are the interface that your program (no matter what programming language it’s in) uses to

why system calls are an amazing interface

System calls are my favorite interface to debug because every program uses system calls.

A few examples of questions that you can ask using system calls:

  • Which files is my program opening or writing to right now? This is surprisingly useful – it has uncovered hidden configuration files and log files for me many times.
  • Which other programs is my program executing? You can see every program that’s being started, and all its arguments
  • Which hosts is my program connecting to (the connect system call). This can also help you see timeouts.

These are all pretty simple questions, but without being able to observe which system calls your program is using, they can be very hard to answer!

How do I see which system calls my program is using?

You use a program called strace on Linux! Or dtruss on OS X.

I have a fanzine about how to use strace, if you are new to it and would like to learn more!

 

Related Posts

OpenTelemetry   Instrumentation  

OpenTelemetry Best Practices #2 Agents, Sidecars, Collectors, Coded Instrumentation

For years, we’ve been installing what vendors have referred to as “agents” that reach into our applications and pull out useful telemetry information from them....

Observability   Monitoring  

APM From a Developer’s Perspective

In twenty years of software development, I did not have the privilege of being on call, of tending to my software in production. I’ve never...

Software Engineering   Monitoring  

What Is Application Performance Monitoring?

Application performance monitoring, also known as APM, represents the difference between code and running software. You need the measurements in order to manage performance....