Guest Josh Clulow joins the podcast hosts, Bryan and Adam, to discuss the 20-year journey of DTrace. They reminisce about stress and deadlines during integration, reflect on the release of Solaris 9, and discuss the development process of DTrace. They also touch on naming conventions, improvements in boot time, and the positive impact of their technology. Other topics include developing meaningful instrumentation for Java, using DTrace for storage performance analysis, and the significance of CVEs and USDT in tracing.
Anonymous tracing during boot and the ability to debug the system in production were crucial aspects of the early development of D-Trace, providing valuable insights and improvements.
The positive response from colleagues and the support received during presentations on the progress of D-Trace highlighted the significance and potential impact of the project.
The importance of working on projects that capture passion and excitement was discussed, with userland tracing and exploring possibilities for replacing instructions with traps being particularly engaging.
The speaker shared their fears and concerns about the progress of D-Trace, highlighting the need for personal inspiration and the dedication to continually drive development and improvement.
The introduction of Mr. Sparkle greatly aided the identification of probes in user space when using D-Trace, allowing for thorough analysis and preventing confusion with trap instructions.
Deep dives
The journey of developing D-Trace
The speaker recounts the early days of working on D-Trace, starting with the decision to focus exclusively on the project for six months. They mention the importance of being able to debug the system in production and the value of anonymous tracing during boot. They also discuss the positive response from others in the organization during a presentation on the progress of D-Trace.
The impact and benefits of D-Trace
The speaker reflects on the significance of D-Trace and its potential impact, noting how it has enabled them to debug real bugs in the system and make improvements. They emphasize the importance of having a safe and reliable system instrumentation framework and the role of anonymous tracing in examining boot processes. The speaker also appreciates the support and enthusiasm shown by their colleagues.
Working on userland tracing among other projects
The speaker discusses working on userland tracing in addition to other projects in the public infrastructure. They mention their interest in structured disassembly and exploring possibilities for replacing instructions with traps in userland. They highlight the importance of working on projects that capture their passion and excitement.
A memorable trip and concerns about unfinished work
The speaker shares their fears of death while hella skiing and expresses concerns about the progress of D-Trace. They recall writing their last will and testament on a whiteboard, outlining priorities for the development of D-Trace in case they were to die. They also discuss the importance of working on things that truly inspire and drive personal interest.
The birth of Mr. Sparkle and the importance of instrumenting user space
Mr. Sparkle was introduced as a way to identify the location of probes in user space when using D-Trace. It allowed for the disassembly of functions to understand function entry and return, as well as the discrete instructions. Mr. Sparkle was used to prevent confusion with trap instructions and to avoid data being mistaken for instructions.
The milestone of instrumenting every instruction in Firefox
The podcast discusses a milestone where it became possible to instrument every instruction in Firefox using D-Trace. Although there were initial failed attempts, eventually, millions of trace points were turned on in Firefox, allowing for performance analysis and debugging.
The importance of open sourcing D-Trace and the growth of the community
The decision to open source D-Trace was crucial in ensuring its survival outside of the original company. By open sourcing D-Trace, it became a contribution to the wider community, allowing for further development and extension. The podcast highlights the community-driven efforts in improving D-Trace, including additions like JSON actions, speculative tracing, and utility stack helpers.
The importance of confidence in using instrumentation
The podcast emphasizes the significance of confidence when using instrumentation tools like D-Trace. The speaker recalls how the safety and robustness of D-Trace allowed them to deploy it without fear of causing major system issues. The confidence bred by D-Trace outweighed any associated limitations. However, the speaker also highlights a crucial lesson learned when it comes to system safety. Simply avoiding harm to the system is not enough; one must also ensure that the system remains alive and can make forward progress.
The evolution of D-Trace and recent innovations
The podcast discusses the evolution of D-Trace and some recent innovations in its use. The speaker mentions the development of USDT, a crate for Rust that allows for statically defined probes in Rust programs. This innovation has been valuable for the Oxide team, who primarily work with Rust. Additionally, the podcast touches on postmortem tracing, which acts as a flight data recorder for programs, providing valuable insights in the event of crashes. The speaker also mentions the importance of interface stability for probes and the need for more typed programmatic access to tracing information.
If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
