Go build -tags dynamic
a story of Kafka, M1 and linking
I just got a new M1 mac, and I have to say, it's a beast! It's great for programming. This was the first time ever that a fully functional IDE like Intellij could be kept open for a day without being charged. I was excited for this machine, but I came to know about some issues. We will discuss one of such issues. This post particularly discusses with running Kafka dependencies in a Golang app.
As part of one of the projects, we run a fairly complicated data ingestion app which polls for particular events and then transforms the same and sends it to Kafka. I tried running this program using a simple go build and was greeted with a mountain of texts which did not make much sense. Parts of which is shared here.
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)Searching for this and with help from colleagues I found an easy way to solve for this issue. Which was to run
brew install openssl
brew install librdkafka
brew install pkg-config
export PKG_CONFIG_PATH="/opt/homebrew/opt/openssl@3/lib/pkgconfig"
go build --tags dynamic main.goI understood most of these commands except the last one. The last one for me did not make much sense and I went on a trail to understand this particular command. Like a seasoned programmer once told me
Never commit a code block that you cannot explain to another developer.
So it was imperative that I went on this journey and understood the problem at a deeper level.
Build Tags
Build tags are an identifier used in a package or file to indicate whether that file should be indicated. Build tags are used in golang with - - tags. It is extremely useful when you are dealing with integration tests or any use cases where certain files need to be used or replaced with other files. For example if you have 3 versions of same function declared in different files and marked the top of the file with // +build tag1 tag2 and need one for different architectures, you will probably need to run it like go build —tags tag1 main.go
If you have two values separated with comma, then both are expected to be present in go build invocation for it to be included as part of go binary. Detailed explanation can be found here.
Dynamic linking
Whenever you try to execute/run a program in a computer there are two steps involved.
Compilation, where a source code is converted to a bunch of object modules
Linking, where a bunch of object modules are combined together to form an executable
There are two types of linking which happens. Static linking which is a process in which the external libraries which are converted to object modules are inserted into the executable during link time itself and before the program is executed. In case of dynamic linking, a pointer to the external file is provided and actual file is copied over to the in memory copy of main file and then ran. There are cons and pros to each approach and would require a detailed blog to explore further. In short, dynamic linking helps updates to external libraries fairly easy while causing side effects like DLL hell where a person updated a library function dynamically linked which results in the broken source code. Static linking is slightly(very slightly since modern cpu, memory buses etc are very fast) faster. In an embedded environment with limited memory, it might make sense to have static linking. A more detailed discussion of static vs dynamic linking can be found here for the curious.
Librdkafka and dynamic linking
Going through the source code of librdkafka which is used by confluent-kafka-go in their producer codes(very clever of them to use this in all languages by having single c file), it seems that what they do is fairly simple.
"static glibc_linux from librdkafka-static-bundle-v1.7.0.tgz"this is an info line from the package which talks about how the linking is done and from where. What happens normally is that a bundled version of librdkafka is used for any high level calls to the Kafka. This would work if you had x86 or amd64 code and would obviously fail if you use any arm64 code. When you use the tag dynamic, the linker links files dynamically based on what is present in the folder rather than blindly believing in the bundled files. A detailed explanation of this process can be found here.
In conclusion, the workaround uses a very specific build constraint using build tags to use a different file from that which is bundled and dynamic linking make sure that the linked file is actually dynamically linked from the user generated file, making sure that the go file actually compiles and you are able to produce messages to kafka using confluent go producer.
Conclusion
I really enjoy learning about the different techniques that people use to solve problems and the different ways that people think. It is amazing to me how much creativity and innovation goes into something that, on the surface, appears to be so simple. Open source and open discussion allows us to understand and replicate the solutions across multiple fields. It is truly a wonderful time to be alive.
On another note, stable diffusion the open source version of DallE is making waves across AI space. It would be interesting to try it out and see what can be done in video space with something like that. I will end the musing with a wonderful quote from a wonderful singer
I see skies of blue
And clouds of white
The bright blessed day
The dark sacred night
And I think to myself
What a wonderful world
Louis Armstrong