Monitoring   Instrumentation  

Instrumentation: system calls: an amazing interface for instrumentation

By Guest Blogger  |   Last modified on December 1, 2018

When you’re debugging, there are two basic ways you can poke at something. You can:

  • create new instrumentation (like “adding print statements”)
  • use existing instrumentation (“look at print statements you already added”, “use Wireshark”)

When your program is already running and already doing some TERRIBLE THING YOU DO NOT WANT, it is very nice to be able to ask questions of it (“dear sir, what ARE you doing”) without having to recompile it or restart or anything.

I think about asking questions of a program in terms of “what interfaces does it have that I can observe?”. Can I tell which system calls that program is using? Can I look at its network requests? Can I easily see database queries?


In this post, we’ll talk about my very favourite interface: system calls

what’s a system call?

Your program does not know how to open files: it doesn’t know what a hard drive is, or how to read a filesystem, or any of that. However, your operating system does know all that stuff!

System calls are the interface that your program (no matter what programming language it’s in) uses to

why system calls are an amazing interface

System calls are my favorite interface to debug because every program uses system calls.

A few examples of questions that you can ask using system calls:

  • Which files is my program opening or writing to right now? This is surprisingly useful – it has uncovered hidden configuration files and log files for me many times.
  • Which other programs is my program executing? You can see every program that’s being started, and all its arguments
  • Which hosts is my program connecting to (the connect system call). This can also help you see timeouts.

These are all pretty simple questions, but without being able to observe which system calls your program is using, they can be very hard to answer!

How do I see which system calls my program is using?

You use a program called strace on Linux! Or dtruss on OS X.

I have a fanzine about how to use strace, if you are new to it and would like to learn more!


Related Posts


Defensive Instrumentation Benefits Everyone

A lot of reasoning in content is predicated on the audience being in a modern, psychologically safe, agile sort of environment. It’s aspirational, so folks...

Instrumentation   Ask Miss O11y  

Ask Miss O11y: How Can I Convince My Organization to Invest in Instrumenting for Observability?

"Dear Miss O11y, I’ve been following Honeycomb for a long time, and I understand where the insights from observability fit in. But larger orgs haven’t...

OpenTelemetry   Monitoring  

Monitoring Unit Tests with OpenTelemetry in .NET

In this post, we’ll look at how you can use OpenTelemetry to monitor your unit tests and send that data to Honeycomb to visualize. It’s...