Skip to content

Commit b3f8916

Browse files
practicalli-johnnypracticalli-johnny
committed
new gitbook: kafka and clojure
- new book structure - deployment script - initial content - work in progress notes
0 parents  commit b3f8916

21 files changed

+716
-0
lines changed

‎README.md

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Kafa and Clojure - Immutable Event Streams
2+
3+
[![GitHub version](https://badge.fury.io/gh/practicalli%2Fspacemacs-gitbook.svg)](https://badge.fury.io/gh/practicalli%2Fspacemacs-gitbook)
4+
5+
![kafka-clojure-logo](images/kafka-clojure-logo.png)
6+
7+
This is an introduction to developing Kafka and Clojure applications, specifically systems built around an immutable event stream.
8+
9+
[![Join the conversation on Gitter](images/gitter-chat-badge-practicalli.png)](https://gitter.im/practicalli/chat?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
10+
11+
This workshop will cover the following topics:
12+
13+
* Understanding Kafka architecture
14+
* Producers and Consumers
15+
* Designing Kafka (topics, partitions consumer groups, etc)
16+
* Replaying events
17+
* Effective Monitoring
18+
19+
20+
## Configuration
21+
22+
> **TODO** work in progress, no git repository available as yet.
23+
24+
[Kafka and Clojure Code Examples](https://github.com/practicalli/kafka-clojure-code-example)
25+
26+
To get a copy of the repository, use the following git clone command which creates a new directory called clojure-webapps-example that contains the cloned code.
27+
28+
```
29+
git clone https://github.com/practicalli/kafka-clojure-code-examples.git`
30+
```
31+
32+
------------------------------------------
33+
34+
This work is licensed under a Creative Commons Attribution 4.0 ShareAlike License (including images & stylesheets).
35+
36+
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" property="dct:title">Kafka and Clojure, Practicalli</span> by <a xmlns:cc="http://creativecommons.org/ns#" href="kafka-and-clojure.practical.li" property="cc:attributionName" rel="cc:attributionURL">John Stevenson</a> is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.<br />Based on a work at <a xmlns:dct="http://purl.org/dc/terms/" href="https://github.com/practicalli/kafka-and-clojure" rel="dct:source">Kafka and Clojure, Practicalli on Gitub</a>.<br />Permissions beyond the scope of this license may be available at <a xmlns:cc="http://creativecommons.org/ns#" href="https://twitter.com/jr0cket" rel="cc:morePermissions">@jr0cket</a>.

‎SUMMARY.md

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Summary
2+
3+
* [Introduction](introduction.md)
4+
* [Kafka and Clojure](introduction/index.md)
5+
* [About this book](introduction/about-this-workshop.md)
6+
* [Troubleshooting](troubleshooting/index.md)
7+
* [Reference](reference/index.md)
8+
* [Contributors](contributors.md)
9+
* [About this workshop](introduction/about-this-workshop.md)
10+
* [Workshop conventions](introduction/workshop-conventions.md)
11+
* [Additional resources](introduction/additional-resources.md)
12+
* [Video discussions](video-discussions.md)
13+
* [Work in progress - messy](work-in-progress.md)

‎book.json

+76
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
{
2+
"gitbook": ">=2.0.0",
3+
"title": "Kafka and Clojure - immutable event stream applicationss",
4+
"description": "Building applications based on immutable event streams using Kafka and Clojure to manage unreasonably large data.",
5+
"structure": {
6+
"readme": "introduction.md"
7+
},
8+
"styles": {
9+
"website": "styles/website.css",
10+
"ebook": "styles/ebook.css",
11+
"pdf": "styles/pdf.css",
12+
"mobi": "styles/mobi.css",
13+
"epub": "styles/epub.css"
14+
},
15+
"plugins": [
16+
"toggle-chapters",
17+
"codeblock-disable-glossary",
18+
"disqus",
19+
"gist",
20+
"exercises",
21+
"quizzes",
22+
"toolbar",
23+
"ga",
24+
"callouts",
25+
"sectionx",
26+
"klipse",
27+
"youtube"
28+
],
29+
"pluginsConfig": {
30+
"callouts": {
31+
"showTypeInHeader": false,
32+
"star": {
33+
"alert": "warning",
34+
"picto": "fa-star"
35+
}
36+
},
37+
"sectionx": {
38+
"tag": "b"
39+
},
40+
"disqus": {
41+
"shortName": "kafka-and-clojure-practicalli"
42+
},
43+
"ga": {
44+
"token": "UA-28030709-9"
45+
},
46+
"toolbar": {
47+
"buttons":
48+
[
49+
{
50+
"label": "Gitter Chat",
51+
"icon": "fa fa-comments-o",
52+
"url": "https://gitter.im/practicalli/chat?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge"
53+
},
54+
{
55+
"label": "GitHub",
56+
"icon": "fa fa-github",
57+
"url": "https://github.com/practicalli"
58+
},
59+
{
60+
"label": "Search page title on StackOverflow",
61+
"icon": "fa fa-stack-overflow",
62+
"url": "http://stackoverflow.com/search?q={{title}}"
63+
}
64+
]
65+
}
66+
},
67+
"links": {
68+
"sidebar": {
69+
"Kafka": "https://kafka.apache.org/",
70+
"Practical.li": "https://practicalli.github.io"
71+
}
72+
},
73+
"variables": {
74+
"host": "practical.li"
75+
}
76+
}

‎deploy-to-gh-pages

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/bin/sh
2+
3+
## A quick hack to deploy a generated website onto a Github repository, to be served as static content by github pages. This is an example of a very bad shell script.
4+
5+
# This should only be run from the root of the gitbook project
6+
# Replace the remote Github URL to use this on a different project
7+
8+
# Trigger a clean and new build
9+
gitbook build .
10+
11+
# Deploy new build
12+
cd _book && rm -rf .git && git init && git add . && git commit -m "initial commit" && git branch -m master gh-pages && git remote add practicalli git@github.com:practicalli/kafka-and-clojure.git && git push -f practicalli gh-pages

‎introduction.md

+81
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Kafka and Clojure - Immutable event streams
2+
3+
>#### Warning::This workshop has only just started to be written
4+
5+
![Kafka](https://kafka.apache.org/images/kafka_diagram.png)
6+
7+
8+
[Kafka](https://kafka.apache.org/) is Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, extremely fast and runs in production in thousands of companies (Braintree).
9+
10+
[Clojure](https://clojure.org/) is a dynamic, general-purpose programming language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for multithreaded programming. Clojure is a compiled language, yet remains completely dynamic – every feature supported by Clojure is supported at runtime. Clojure provides easy access to the Java frameworks, with optional type hints and type inference, to ensure that calls to Java can avoid reflection.
11+
12+
## Why Kafka and Clojure
13+
14+
* Easy interaction with and well-maintained libraries for other JVM-based systems (e.g. Kafka and ElasticSearch)
15+
* Concurrency and IPC capabilities
16+
* Clojure's sequence abstractions simplified the code and made testing easy
17+
18+
19+
## Tutorials
20+
* [Apache Kafka - quickstart](https://kafka.apache.org/quickstart)
21+
* [Part 1: Apache Kafka for beginners - What is Apache Kafka - CloudKarafka](https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html)
22+
* [Cloudurable Kafka Tutorial](http://cloudurable.com/blog/kafka-tutorial/index.html) - current tutorial for Meet-a-mentor study group
23+
* [Apache Kafka Tutorial - TutorialsPoint](https://www.tutorialspoint.com/apache_kafka/index.htm)
24+
* [Kafka in Clojure](https://techblog.roomkey.com/posts/clojure-kafka.html)
25+
* [Writing a Kafka Producer and High Level Consumer in Clojure](https://wtfleming.github.io/2015/01/14/kafka-clojure-producer-consumer/)
26+
* [Hello World Kafka](https://github.com/SubhaSingh/hello_world_kafka)
27+
*
28+
29+
30+
## Libraries and projects
31+
* [franzy](https://github.com/clj-kafka/franzy) - suite of Clojure libraries for Apache Kafka. It includes libraries for Kafka consumers, producers, partitioners, callbacks, serializers, and deserializers.
32+
* [Kafka Streams Clojure](https://github.com/bobby/kafka-streams-clojure) - Clojure transducers interface to Kafka Streams. This combo provides the best of both worlds for building streaming applications on Kafka with Clojure
33+
* [milena](https://github.com/dvlopt/milena) - This Kafka client library allows the user to exchange records while speaking clojure
34+
35+
36+
## Presentations on Kafka
37+
* [Introduction to Apache Kafka by James Ward](https://www.youtube.com/watch?v=UEg40Te8pnE)
38+
* [Building a data pipeline with Clojure and Kafka](https://speakerdeck.com/davidpick/building-a-data-pipeline-with-clojure-and-kafka)
39+
* [Lisp in the Machine - Clojure at Braintree](https://vimeo.com/138748699)
40+
* [Ployconf 15 - The LISP in the Machine A Clojure experience report from Braintree / Joe Nash](https://www.youtube.com/watch?v=0D3jev1E5ks)
41+
* [One Million Clicks per Minute with Kafka and Clojure - Devon Peticolas](https://www.youtube.com/watch?v=VC_MTD68erY)
42+
* [From REST to CQRS with Clojure, Kafka, & Datomic - Bobby Calderwood](https://www.youtube.com/watch?v=qDNPQo9UmJA&t=8s)
43+
* [David Pick - Building a Data Pipeline with Clojure and Kafka](https://www.youtube.com/watch?v=6xlyWjqFDWs)
44+
* [Lambda Days 2018 - Andrea Crotti - Tame Kafka streams with Clojure](https://www.youtube.com/watch?v=OC2KVaLQihs)
45+
* ["Commander: Better Distributed Applications through CQRS and Event Sourcing" by Bobby Calderwood](https://www.youtube.com/watch?v=B1-gS0oEtYc)
46+
* [Streaming Data Platforms & Clojure with Derek Troy-West](https://www.youtube.com/watch?v=4sUaF4m5TWI)
47+
* [Clojure at 4,000 msg/s-What We Learned, Loved, and Loathed - Nathan Barnett](https://www.youtube.com/watch?v=zwuFJovzdHg)
48+
* [Managing One of the World's Largest Clojure Code Bases - Donevan Dolby](https://www.youtube.com/watch?v=iUC7noGU1mQ)
49+
* [Reducing Microservice Complexity with Kafka and Reactive Streams - by Jim Riecken](https://www.youtube.com/watch?v=k_Y5ieFHGbs&t=2s)
50+
* [PolyConf 15: Distributed systems the easy way with Clojure and Mesos / Pierre-Yves Ritschard](https://www.youtube.com/watch?v=Tz-wwjhqEEM)
51+
52+
53+
<div style="width:80%; margin:auto;">
54+
This work is licensed under a Creative Commons Attribution 4.0 ShareAlike License (including images & stylesheets).
55+
56+
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" property="dct:title">Spacemacs, Practicalli</span> by <a xmlns:cc="http://creativecommons.org/ns#" href="spacemacs.practical.li" property="cc:attributionName" rel="cc:attributionURL">John Stevenson</a> is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.<br />Based on a work at <a xmlns:dct="http://purl.org/dc/terms/" href="https://github.com/practicalli/spacemacs" rel="dct:source">Spacemacs, Practicalli on Gitub</a>.<br />Permissions beyond the scope of this license may be available at <a xmlns:cc="http://creativecommons.org/ns#" href="https://twitter.com/jr0cket" rel="cc:morePermissions">@jr0cket</a>.
57+
</div>
58+
59+
60+
61+
> ####TODO::Introductory video
62+
> Why kafka and Clojure. What is an immutable event stream
63+
64+
<iframe width="800" height="420" src="https://www.youtube.com/embed/Uuwg-069NYE?ecver=1" frameborder="0" allowfullscreen></iframe>
65+
66+
<table>
67+
<tr>
68+
<td>
69+
<a href="why-spacemacs/index.html"><img src="images/button-why-spacemacs.png" /></a>
70+
</td>
71+
<td>
72+
<a href="before-you-start/index.html"><img src="images/button-before-you-start.png" /></a>
73+
</td>
74+
<td>
75+
<a href="install-spacemacs/index.html"><img src="images/button-install-spacemacs.png" /></a>
76+
</td>
77+
<td>
78+
<a href="create-a-project/index.html"><img src="images/button-code-with-clojure.png" /></a>
79+
</td>
80+
</tr>
81+
</table>

‎introduction/about-this-workshop.md

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# About this workshop
2+
3+
This is an introduction to developing applications using Kafka and Clojure, specifically applications that use an immutable event stream.
4+
5+
6+
## Topics covered in Workshop
7+
8+
This workshop will cover the following topics:
9+
10+
*
11+
*
12+
*
13+
*
14+
*
15+
*

‎introduction/additional-resources.md

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Additional Resources
2+
3+
**Spacemacs**
4+
* [Spacemacs Website](http://spacemacs.org/)
5+
* [Spacemacs ABC](https://www.youtube.com/playlist?list=PLrJ2YN5y27KLhd3yNs2dR8_inqtEiEweE) - screencast walk-through of features in alphabetical order
6+
* [SimpleTutorials: Spacemacs](https://simpletutorials.com/c/2883/Spacemacs)
7+
8+
**Clojure Getting Started**
9+
* [ClojureBridge London curiculum](http://clojurebridgelondon.github.io/curriculum)
10+
* [Clojure through code](https://github.com/practicalli/clojure-through-code/tree/drafts) - many examples of basic Clojure code
11+
* [Getting started with Clojure](http://jr0cket.co.uk/slides/getting-started-with-clojure.html) - a general guide to Clojure (work in progress)
12+
* [4Clojure](https://www.4clojure.com/) exercises & [4Clojure Google group](https://groups.google.com/forum/#!forum/4clojure)
13+
* [Clojurescript Koans](http://clojurescriptkoans.com/) or [Lighttable Koans](https://github.com/practicalli/lighttable-koans) - exercises to help you discover Clojure
14+
15+
**Clojure Reference**
16+
* [Clojure.org](http://clojure.org), [features](http://clojure.org/features) and [rational](http://clojure.org/rationale)
17+
* [Clojure documentation](http://clojure.org/documentation)
18+
* [Clojure cheetsheet](http://clojure.org/cheatsheet)
19+
* [Official Google group](https://groups.google.com/forum/#!forum/clojure)
20+
* [CrossClj](http://crossclj.info/) - cross-referencing the Clojure ecosystem
21+
22+
**Tooling**
23+
* [Leiningen](http://leiningen.org/), [tutorial](https://github.com/technomancy/leiningen/blob/stable/doc/TUTORIAL.md), [faq](https://github.com/technomancy/leiningen/blob/stable/doc/FAQ.md), [plugins list](https://github.com/technomancy/leiningen/wiki/Plugins) & [sample project file](https://github.com/technomancy/leiningen/blob/stable/sample.project.clj)
24+
25+
26+
**Books & Tutorials**
27+
* [Practicalli, Clojure](http://practicalli.github.io/clojure/)
28+
* [Practicalli, Clojure WebApps](http://practicalli.github.io/clojure-webapps/)
29+
* [Practicalli, Clojurescript](http://practicalli.github.io/clojurescript/)
30+
* [Clojure cookbook](https://github.com/clojure-cookbook/clojure-cookbook)
31+
* [Brave Clojure](http://www.braveclojure.com/) - a Clojure tutorial (using Emacs)
32+

‎introduction/contributing.md

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Contributing to this workshop
2+
3+
Contributions to the content of this workshop are welcome, either via
4+
5+
* Disqus comments on each page
6+
7+
* Issues or Pull requests (once the markdown is published)
8+
9+
* [The Gitter chat room](https://gitter.im/practicalli/chat?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
10+
11+
[![Join the conversation on Gitter](/images/gitter-chat-badge-practicalli.png)](https://gitter.im/practicalli/chat?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
12+
13+
By submitting content ideas and corrections you are agreeing they can be used in this workshop under the [Creative Commons Attribution ShareAlike 4.0 International license](http://creativecommons.org/licenses/by-sa/4.0/). Attribution will be detailed in the [contributors](contributors.html) section of this workshop.
14+
15+
Any contributors will be noted in the [contributors](contributors.html) section.
16+
17+
Thank you.

‎introduction/index.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Introduction to Kafka and Clojure

‎introduction/sample-configuration.md

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Sample Spacemacs Configuration
2+
3+
Any customisations made to the standard Spacemacs install are contained in the Github repository [Spacemacs configuration](https://github.com/practicalli/spacemacs-config)
4+
5+
Either [browse the repository online](https://github.com/practicalli/spacemacs-config) or use your favorite git client or the following git clone command:
6+
7+
`git clone https://github.com/practicalli/spacemacs-config.git`
8+
9+
The above command creates a new directory called `spacemacs-config` that contains the configuration examples from this book, all in one place.
10+
11+
You can use **ediff** in Spacemacs to compare your configuration files with the examples.

‎introduction/workshop-conventions.md

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Workshop Conventions
2+
3+
## Keybinding conventions
4+
5+
General menu and commands
6+
7+
| Spacemacs | Major mode | Normal | Insert | Command |
8+
|---------------------|-----------------------|------------------------|------------------------|-----------------------|
9+
| `SPC` - global menu | `,` - major mode menu | Evil (Vim) normal mode | Evil (Vim) insert mode | Name of Emacs command |
10+
11+
12+
Vim Mode specific keybindings
13+
14+
| Normal | Insert | Visual | Motion | Lisp | Iedit | Evilified | Emacs |
15+
|------------------|-----------------------|--------------------|--------|--------------------|-------------------|-----------|---------------------------|
16+
| The default mode | Mode for writing text | Mode for selection | ? | Structured Editing | Text manipulation | | Classic Emacs keybindings |
17+
| | | | | | | | |
18+
19+
20+
21+
### Spacemacs - `SPC`
22+
23+
Spacemacs provides a global menu where you can access commands using a mnemonic, text based menu. Its the recommended starting point for those new to Spacemacs.
24+
25+
Access the global menu via `SPC` in Evil Normal mode or via `M-m` in all other modes
26+
27+
> ####Info
28+
> The use of `SPC`
29+
30+
31+
### Major Mode - `,`
32+
33+
Access features specific to the current major mode.
34+
35+
For example when you have a Clojure file or REPL buffer open you can access a Clojure specific menu using `M-RET` in Emacs mode or `,` in Vim normal mode.
36+
37+
Having a major mode specific menu removes the need to use `SPC m` in Vim normal mode or `M-m` in Emacs mode.
38+
39+
40+
### Evil (normal mode)
41+
42+
Vim arranges keybindings across multiple modes, reducing the number of key presses required to carry out commands normal mode specific short-cuts to access Spacemacs features
43+
44+
45+
### Command - `SPC SPC`
46+
47+
Every menu item and keybinding in Spacemacs is calling a function, collectively called commands.
48+
49+
Use commands with `SPC SPC` in Vim normal mode or `M-x` in any mode.
50+
51+
Use a command when there is no menu item or keybinding.
52+
53+
## Classic Emacs Keybindings
54+
55+
The keybindings used in Emacs (holy mode) are also available, many of which start with `C-c` or `C-x`. If you are familiar with these then its fine to stick with them. However, if you have trouble learning or remembering them, then switch to the Spacemacs mnemonic alternatives.

‎mind-maps/Kafka Part 1.pdf

84.2 KB
Binary file not shown.

‎mind-maps/Kafka part 3.pdf

79.6 KB
Binary file not shown.

‎study-group-slack-notes.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Notes from Slack study group

‎styles/ebook.css

Whitespace-only changes.

‎styles/epub.css

Whitespace-only changes.

‎styles/mobi.css

Whitespace-only changes.

‎styles/pdf.css

Whitespace-only changes.

0 commit comments

Comments
 (0)