Click here to check if anything new just came in.
January 23 2012
January 20 2012
January 15 2012
October 17 2011
Why Chef?
Let me tell you a little more about how wonderful Chef is, then I will show you briefly some technical highlights, and finally I will clear up some Chef myths.
The Top 5 Reasons to use Chef
- Writing reams of documentation sucks. Chef drastically reduces the amount of documentation you have to write.
- Bash doesn't scale. Seriously.
- Technical Awesomeness
- Chef grows with you
- You can stop reinventing the wheel
Write Less Documentation
basic sendmail configuration with chef
Bash Doesn't Scale
Bash is a wonderful thing, but like all UNIX tools, it is fundamentally limited by design. Bash doesn't have a code reuse mechanism more powerful than functions. It uses a single global namespace. These and other limitations have made it hard for us sysadmins to reuse and generalize our shell scripts across different distributions, let alone different versions of *nix. Lastly, it is quite difficult to write shell scripts that are idempotent, that is, they have the same effect whether run once or 100 times. This is particularly true when manipulating configuration files. Let's take a look at how you would configure /etc/sudoers with chef.
All the variables in this template are passed in from this recipe. If you think you can replicate this template using sed, please submit it as a comment so we can compare results.
Last but not least, Bash can be just as unreadable and obfuscated as the darkest perl code. With Chef you have a high level, intuitive DSL for describing your configuration. Remember what I said about needing less documentation?
Technical Awesomeness
NOSQL FTW
One of the virtues that many *nix tools share is that they store their configurations in text files rather than binary formats or in a database. Chef stores your system configurations in text and in a database. It accomplishes this by using the document-oriented database, CouchDB. This makes the configuration searchable and higher performance. One of my annoyances with nagios is that I have to restart it every time I change any service or host. There are no annoying restarts with Chef. Each Configuration change you make takes effect instantly. Further you can use your favorite text editor and the command line to make configuration changes. Finally, using a database means that Chef is easily accessed through a GUI. We sysadmins often look down on GUIs as crutches for the Windows crowd. The Chef web UI is excellent for visualizing your own infrastructure, something that is hard to do while staring at plain text.
Knowing is Half the Battle
Chef uses Ohai to collect data about your system. Your recipes can access these attributes and make decisions based on them. For example, you can determine which version of Red Hat you are using simply by looking up the value of node['platform_version']. You don't have to cat | grep | awk to find out which release you are on.
Ohai makes cookbooks more dynamic and able to support different distributions. As we will see later, this is one of the reasons there are so many high-quality cookbooks available.
Search
Search is a feature in Chef Server that allows you to query the configuration information of all other servers and of globally-defined databags. This allows you to do things like configure clusters where a member of cluster needs to know not only about its own configuration but about the configurations of the other members of the cluster.
Example of using search with data bags. For the full recipe go here.
Knife
Knife is one of the truly great command line tools. It is your primary mechanism for interacting with the chef-server. Knife shares many usage patterns with git. If you love git, you'll love knife.
shef
shef works the way you work, in an iterative manner. Most of us system administrators are self-taught and we learn best by doing. Fire up shef and you can on the fly play with attributes and create recipes. Further, you can connect to your server and download the cookbooks.
Chef Grows with You
Chef uses pure Ruby as its configuration language, not a shackled subset of ruby, nor yet another custom configuration language. You only have to learn a small amount of ruby to get started with chef. Once you get beyond the basics of Chef, you can go farther with Ruby. Many of you are grumbling now because ruby sucks to compared to Perl/Python/TCL/<insert-interpeted-language-here>. Well, Ruby may pale in comparison to Python but it is still a powerful, full-featured language.
Just like Perl, there is a lot of dark magic inside Ruby. It can't be used carelessly or it will bite you in the ass. Unlike Python, Ruby does not make it difficult to do the wrong thing.
You can stop reinventing the wheel
Until Chef, we sysadmins did not have a truly modular way to abstract and share our system configurations. Please stop reading and browse http://community.opscode.com/cookbooks. Later on, you should look at the code in https://github.com/opscode/cookbooks. You will discover that some turncoat sysadmins are giving away our trade secrets. Now is the time for you to do the same. In fact, you can rip out a large chunk of your shell scripts and replace them with Chef cookbooks. You will find that many existing recipes meet your requirements and you can easily add new recipes to existing cookbooks for your unmet requirements.
Recently, I replaced a 500 line shell script with 7 cookbooks, 5 reused from community.opscode.com and 2 new ones written from scratch. In the process of replacing the shell script I actually wrote 4 cookbooks that unwittingly duplicated the functionality of existing cookbooks.
The emergence of the cloud-computing, whether on the public cloud such as EC2 or internally with openstack, means that we systems administrators will have to manage at least 10 times more systems. We have to become 10 times more productive. We have to shift from "managing servers" to "managing configurations." Chef is one of key tools to accomplishing this.
One Big Fat Chef Caveat
Chef makes good sysadmins more productive. It does not turn junior sysadmins into experts. In fact, it makes them more dangerous. As I overheard on IRC one day, "Chef is to sysadmins what C++ is to software engineering." This is very true. You can automate system configurations with off-the-shelf cookbooks but you have to understand exactly what they do and test them very rigorously.
Chef Myths
There are a few myths about Chef floating around the intertubes that need to be exploded. The biggest one is that you have to be a professional programmer to use Chef. This is untrue for a couple of reasons, including the assumption that system administrators don't program already. Writing complex shell scripts is programming. Creating Apache rewrite rules is quite close to programming. Further, Chef doesn't require you to be a Rubyist to get started, just as you don't have to be a Bash Hacker to use the command line. Take a look at this tutorial and you will see what I mean.
Comments
Amazon kindle source code
Source Code Notice
Amazon is pleased to make available to you for download an archive file of the machine readable source code ("Source Code") corresponding to modified software packages used in the Kindle device. By downloading the Source Code, you agree to the following:
AMAZON AND ITS AFFILIATES PROVIDE THE SOURCE CODE TO YOU ON AN "AS IS" BASIS WITHOUT REPRESENTATIONS OR WARRANTIES OF ANY KIND. YOU EXPRESSLY AGREE THAT YOUR USE OF THE SOURCE CODE IS AT YOUR SOLE RISK. TO THE FULL EXTENT PERMISSIBLE BY APPLICABLE LAW, AMAZON AND ITS AFFILIATES DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. AMAZON AND ITS AFFILIATES WILL NOT BE LIABLE FOR ANY DAMAGES OF ANY KIND ARISING FROM THE USE OF THE SOURCE CODE, INCLUDING, BUT NOT LIMITED TO DIRECT, INDIRECT, INCIDENTAL, PUNITIVE, AND CONSEQUENTIAL DAMAGES.
Click on the links below to download an archive file of the Kindle machine readable Source Code:
Kindle
Wi-Fi, 6" E Ink Display
Kindle (Latest Generation, Wi-Fi, 6")
Kindle 3G (Latest Generation, Free 3G + Wi-Fi, 6")
- Kindle_src_3.0_515460094.tar.gz
- Kindle_src_3.0.1_525120101.tar.gz
- Kindle_src_3.1_558700031.tar.gz
- Kindle_src_3.2_572340009.tar.gz
- Kindle_src_3.2.1_576290015.tar.gz
Kindle (2nd Generation, Free 3G)
- Kindle_src_2.2_375490138.tar.gz
- Kindle_src_2.2.1_375890239.tar.gz
- Kindle_src_2.2.2_388760001.tar.gz
- Kindle_src_2.3_399380047.tar.gz
- Kindle_src_2.3.3_431100003.tar.gz
- Kindle_src_2.5.2_490480060.tar.gz
- Kindle_src_2.5.4_501100064.tar.gz
- Kindle_src_2.5.7_550650009.tar.gz
- Kindle_src_2.5.8_555370010.tar.gz
Kindle (2nd Generation, U.S. Wireless)
- Kindle_src_2.0_291330095.tar.gz
- Kindle_src_2.0.1_303870012.tar.gz
- Kindle_src_2.0.2_309510017.tar.gz
- Kindle_src_2.0.3_327610024.tar.gz
- Kindle_src_2.0.4_353720025.tar.gz
- Kindle_src_2.3_399380047.tar.gz
- Kindle_src_2.3.3_431100003.tar.gz
- Kindle_src_2.5.2_490480060.tar.gz
- Kindle_src_2.5.4_501100064.tar.gz
- Kindle_src_2.5.7_550650009.tar.gz
- Kindle_src_2.5.8_555370010.tar.gz
Kindle DX (Free 3G)
- Kindle_src_2.5.5_495460008.tar.gz
- Kindle_src_2.5.7_550650009.tar.gz
- Kindle_src_2.5.8_555370010.tar.gz
Kindle DX (Free 3G)
- Kindle_src_2.3_399380047.tar.gz
- Kindle_src_2.3.3_431100003.tar.gz
- Kindle_src_2.5.2_490480060.tar.gz
- Kindle_src_2.5.4_501100064.tar.gz
- Kindle_src_2.5.7_550650009.tar.gz
- Kindle_src_2.5.8_555370010.tar.gz
Kindle DX (U.S. Wireless)
- Kindle_src_2.1_337560062.tar.gz
- Kindle_src_2.1.1_351050064.tar.gz
- Kindle_src_2.3_399380047.tar.gz
- Kindle_src_2.3.3_431100003.tar.gz
- Kindle_src_2.5.2_490480060.tar.gz
- Kindle_src_2.5.4_501100064.tar.gz
- Kindle_src_2.5.7_550650009.tar.gz
- Kindle_src_2.5.8_555370010.tar.gz
Kindle (1st Generation)
- Kindle_src.1.0.0.292
- Kindle_src_1.0.4_144750018
- Kindle_src_1.0.8_164820023
- Kindle_src_1.1_177110002
- Kindle_src_1.1.1_238920002
- Kindle_src_1.2_299870016
Comments
October 14 2011
Dolly Drive: Time Machine in the Cloud
As many of the Mac AppStorm writers will tell you, backup is important! It is the single thing that is protecting you from massive data loss, hours of frustration and lots of hair pulling.
With the advent of Leopard, Apple released a built-in backup utility that makes backup a breeze, called Time Machine. However, Time Machine was developed for local use only. It will backup to a Firewire or USB hard drive plugged directly into your computer as well as a Time Capsule device on your local Wifi network. While that is a very good thing, natural disasters do occur, as does theft and simple hard drive failure that can put your backup at risk. What if you could use Time Machine to backup to the cloud?
Introducing Dolly Drive
Dolly Drive does just that. It enables you to use Time Machine to backup to a cloud service, called Dolly Grid.

Backing up to Dolly Grid
Backing up using Dolly Drive just requires a small application that changes a few things about your Time Machine settings. Instead of backing up to a local hard disk or Time Capsule on your local network, it creates a backup that is transferred up to the Dolly Grid.

Dolly Drive main window

Dolly Drive Backup Status
Now one thing that must be remembered is the slowness that is associated with online backup. Whether you use Dropbox, CrashPlan or Dolly Drive, your backups are going to take a bit longer than they would if they were backing up to a local hard disk. However, the benefits (protection against theft, hard drive failure or natural disaster) often outweigh the downside of slower backup.
Cloning With Dolly Clone
Once you have your time machine backing up to the cloud, what are you going to do with the hard drive that is sitting idle besides your computer? Use it as a local backup of course! With most of your data secured online, it can takes hours to download your data to get going again after a hard drive failure or loss of some kind. Having a local backup as well a cloud backup will help you get up and running again in a matter of minutes instead of hours.
Since Dolly Drive takes up your Time Machine capabilities, (Apple doesn’t allow for two different Time Machine instances to exist on one Mac at the same time) you will need to use a cloning utility instead. Recently, Dolly Drive added cloning capabilities right inside their application under the name “Dolly Clone.”

Dolly Clone, selecting a source
Dolly Clone is about as simple as it gets. You pick what you want backed up and then which drive it should be cloned to. Then you can chose to have Dolly Clone wipe the backup destination and start fresh, or have it smartly update the drives to be clones of each other. The latter is done by determining the differences between the two drives and then adjusting the destination drive to match the original.
Pricing Online Backup
Dolly Drive is a subscription service (with Dolly Clone being a free download for everyone). They have a few different plans starting at $5/month for 50GB, going up to $10/month for 250GB and even $55/month for 2TB of storage (there are discounts available if your pay in advance). Each plan comes with an extra 5GB per month that you remain a customer. Since Time Machine backups continuously expand, it’s a great bonus to using Dolly Drive.
The two main competitors to Dolly Drive appear to be CrashPlan and Backblaze. However, these don’t utilize the built-in Time Machine system to backup. They each charge $5/month for unlimited backup. It’s important to note though that restoring from these services generally requires logging onto their website and downloading a .zip file. This is much less fluid than using Time Machine to connect to your Dolly Drive backup and restoring from there.
Conclusion
Dolly Drive for Lion, at the time of writing is still in Beta. There are a few bugs that should be fixed with Lion’s 10.7.2 backup, according to Dolly Drive. However it worked splendidly for me.
It is stuck with the normally slow internet backup problem that all of its competitors also face. With a normal home connection, the Internet isn’t really fast enough to match local backup speeds. While it isn’t Dolly Drive’s fault, it is something to think about if you plan to start backing up terabytes of data.
Because it is using Time Machine to backup, there isn’t a way to access your files on a mobile device or different computer, even if your files are located in the cloud.
Should you start using Dolly Drive for cloud backup? I would say yes if you haven’t ever tried online backup. Being so deeply integrated with the Mac operating system is fantastic. I found their support to be exceptional as well. If you are already backed up with another online backup service, I would be a bit weary. This is mainly due to the amount of time that it would take to get all of your data in the cloud again.
Do you use an online backup service? Have you tried out Dolly Drive? Let us know in the comments!
October 04 2011
Homemade GPS Receiver
Homemade GPS Receiver
Back to projects
Pictured above is the front-end, first mixer and IF amplifier of an experimental GPS receiver. The leftmost SMA is connected to a commercial antenna with integral LNA and SAW filter. A synthesized first local oscillator drives the bottom SMA. Pin headers to the right are power input and IF output. The latter is connected to a Xilinx FPGA which not only performs DSP, but also hosts a fractional-N synthesizer. More on this later.
I was motivated to design this receiver after reading the work [1] of Matjaž Vidmar, S53MV, who developed a GPS receiver from scratch, using mainly discrete components, over 20 years ago. His use of DSP following a hard-limiting IF and 1-bit ADC interested me. The receiver described here works on the same principle. Its 1-bit ADC is the 6-pin IC near the pin headers, an LVDS-output comparator. Hidden under noise but not obliterated in the bi-level quantised mush that emerges are signals from every satellite in view.
All GPS satellites transmit on the same frequency, 1575.42 MHz, using direct sequence spread spectrum (DSSS). The L1 carrier is spread over a 2 MHz bandwidth and its strength at the Earth's surface is -130 dBm. Thermal noise power in the same bandwidth is -111 dBm, so a GPS signal at the receiving antenna is ~ 20 dB below the noise floor. That any of the signals present, superimposed on one another and buried in noise, are recoverable after bi-level quantisation seems counter-intuitive! But the magic of DSSS makes it possible, of course.
GPS relies on the auto- and cross-correlation properties of pseudo-random sequences called Gold Codes to separate signal from signal and signal from noise. Each satellite has a unique sequence. Detection is possible above a minimum signal-to-noise ratio (SNR). All uncorrelated signals are noise, including those of other satellites and quantisation errors. Hard-limiting (1-bit ADC) actually only degrades SNR by less than 3 dB, a price worth paying to avoid hardware AGC.
Architecture
My homemade GPS receiver has two printed circuit boards: the front-end depicted above and an FPGA board from an earlier frequency synthesizer project which was not designed to be reused in this way. The FPGA is controlled by "C" software running on a Windows PC connected to the FPGA via a Xilinx Platform USB JTAG cable. JTAG is slow but then so are GPS data rates. The FPGA performs real-time signal processing autonomously. Its principle roles are: synthesizing the first local oscillator and IF DSP. Computational "heavy-lifting" such as FFT-based search and solving for user position are performed by the PC.
This is a four-channel GPS receiver, which means it can track four satellites simultaneously. A minimum of four are required to solve for user position and receiver clock bias. Inside the FPGA, the tracking module is instantiated four times. In principle, more channels could easily be added simply by changing the size of a Verilog instance array; however, the Spartan 3 XC3S400 is almost 100% full. It might be possible to track more satellites by multiplexing channels between satellites; but this has not been explored.
The 1575.42 MHz L1 carrier is down-converted to a first IF (intermediate frequency) of 22.5 MHz and amplified by analogue means. Subsequently, all IF and baseband signal processing is DSP in the FPGA. Each of the four channels has digital PI controllers to track carrier and code phase. The 50 bps NAV data is collected in four FIFO memories. These are polled via JTAG by "C" software on the PC which performs parity checks, identification and decoding of subframes. Once the necessary ephemeris is collected, a snapshot is taken of certain internal counters in the FPGA. From this, four times of transmission are computed to a precision of ± 0.1 µs.
Front-end
Signal processing up to and including the hard-limiter:
The LMH7220 comparator has a maximum input offset voltage of 9.5mV. Amplified thermal noise must comfortably exceed this to keep it toggling. Weak GPS signals only influence the comparator near zero crossings! They are "sampled" by the noise! To estimate noise level at the comparator input we tabulate gains, insertion losses and noise figures:
LNA SAW Coax RF Mixer IF Overall system noise figure Gain +28.5 -1.5 -3.9 +19.0 -6 NF 0.8 1.5 3.9 3.3 6 7 1 dB
In-band noise at the mixer output is -174+1+28.5-1.5-3.9+19-6+10*log10(2.5e6) = -73 dBm or 51µV RMS. The mixer is resistively terminated in 50-ohms and the stages thereafter work at higher impedance. The discrete IF strip has an overall voltage gain of 800 so the comparator input level is 40mV RMS.
The LMH7220 adds 59 dB of gain making a total of 116 dB for the whole IF. Deploying so much gain at one frequency was a risk. To minimise it, balanced circuitry over a solid ground plane was used and screened twisted-pair carries the output to the FPGA. The motivation was simplicity, avoiding a second conversion. In practice, the circuit is stable, so the gamble paid-off.
Active decoupler Q1 supplies 5V for the remote LNA. MMIC amplifier U2 provides 19 dB gain (not at IF!) and ensures low overall system noise figure, even if long antenna cables are used. L1 and L2 are hand-wound microwave chokes with very high self-resonant frequency, mounted perpendicular to one another and clear of the ground plane. Wind 14 turns, air-cored, 1mm inside diameter from 7cm lengths of 32swg enameled copper wire. Checked with the tracking generator on a Marconi 2383 SA, these were good to 4 GHz.
The Mini-Circuits MBA-15L DBM was chosen for its low 6 dB conversion loss at 1.5 GHz and low 4 dBm LO drive requirement. R9 terminates the IF port.
Three fully-differential IF amplifier stages follow the mixer. Low-Q parallel tuned circuits strung between collectors set the -3 dB bandwidth around 2.5 MHz and prevent build-up of DC offsets. L4, L5 and L6 are screened Toko 7mm coils. The BFS17 was chosen for its high (but not too high) 1 GHz fT. Ie is 2mA for lowest noise and reasonable βre.
The 1st IF is digitally down-converted to 2.5 MHz by under-sampling at 10 MHz in the FPGA. 2.5 MHz lies at the centre of the 5 MHz Nyquist bandwidth. Alternative 1st IF frequencies are 12.5, 17.5 and 27.5 MHz. There is a trade-off between image problems at lower and available BFS17 gain at higher frequencies. Both 22.5 MHz and 27.5 MHz have been tried successfully. 17.5 and 27.5 MHz produce spectrum inversion at the 2nd IF.
Search
Signal detection entails resolving three unknowns: what satellites are in view, their Doppler shifts and code phases. A sequential search of this three-dimensional space from a so-called "cold start" could take many minutes. A "warm start" using almanac data to predict positions and velocities still requires a code search. All 1023 code phases must be tested to find the maximum correlation peak. Calculating 1023 correlation integrals in the time-domain is very expensive and redundant. This GPS receiver uses an FFT-based algorithm that tests all code phases in parallel. From cold, it takes 2.5 seconds on a 1.7 GHz Pentium to measure signal strength, Doppler shift and code phase of every visible satellite.
With over bar denoting conjugation, the cross-correlation function y(Τ) of complex signal s(t) and code c(t) shifted by offset Τ is:

The Correlation Theorem states that the Fourier transform of a correlation integral is equal to the product of the complex conjugate of the Fourier transform of the first function and the Fourier transform of the second function:
FFT(y) = CONJUGATE(FFT(s)) * FFT(c)
Correlation is performed at baseband. The 1.023 Mbps C/A code is 1023 chips or 1ms long. FFT length must be a multiple of this. The sampling rate and FFT length chosen give a 500 Hz bin size. 21 Doppler shifts are tested by rotating the frequency domain data, one bin at a time, up to ±10 bins = ±5 KHz. Rotation can be applied to either function.
The 22.5 MHz 1st IF from the 1-bit ADC is under-sampled by a 10 MHz clock in the FPGA, down-converting it to a 2nd IF of 2.5 MHz. Although not shown above, the samples are held in a FIFO memory before JTAG transfer. In the PC, the 2nd IF is converted to complex baseband (IQ) using quadrature local oscillators. The mixers are simple XOR gates, since the signals are all bi-level. Complex baseband is transformed to the frequency domain by a forward FFT which need only be computed once.
A forward FFT is also required for each satellite's C/A code. Memory permitting, these could be pre-calculated; but generation-on-demand is not that expensive. The 1.023 Mbps code rate is generated by a numerically-controlled-oscillator (NCO) phase accumulator. Unlike 2.5, which is precisely a quarter, 1.023 MHz is not a sub-harmonic of the sampling rate. Consequently, the NCO output has fractional spurs. The number of samples per code chip dithers between 9 and 10.
Processing time is dominated by the inner-most loop which performs shifting, conjugation, complex multiplication and one inverse-FFT per satellite-Doppler test.
There is significant scope for optimisation in the current code.
At 10 MHz sampling rate, code phase is resolved to ~100ns, which seems a little extravagant.
Typical CCF output is illustrated below:

Calculating peak to average power over this data gives a good estimate of SNR and is used to find the strongest signal(s).
The following were received at 20:14 GMT on 4 March 2011 in Cambridge, UK with the antenna on an outside North-facing window ledge:
PRNNAVSTARDoppler (Hz)Code PhaseSNR 9 33 1500 2.495.3 1757 500364.598.4 2253 1000844.754.1 2727 0770.053.8 2848-3000103.999.1
From northern latitudes, more GPS satellites will generally be found in the southern sky i.e. towards the equator.
Tracking
Having scanned the sky and selected the strongest signal, the next step is locking on, tracking it and demodulating the 50 bps NAV data. This requires two inter-dependent phase locked loops (PLLs) to track code and carrier phase. These PLLs must operate in real-time and are implemented as DSP functions in the FPGA. PC software has a supervisory role: deciding which satellites to track, monitoring the lock status and processing the received NAV data.The tracking PLLs are good at maintaining lock, because they have very narrow loop bandwidths; however, this same characteristic makes them poor at acquiring lock without help. A PLL cannot "see" beyond loop bandwidth to capture anything further away. Initial phases and frequencies must be preset to the measured code phase and Doppler shift of the target satellite. This is orchestrated under PC control. The loops should be in-lock from the outset and remain so. NCO dither is seemingly not a problem.
PC processing time varies and there is latency in JTAG communications; however, code phase is measured relative to the 2ms sample. To mark this reference point, the code NCO in the FPGA is reset at the start of sampling and accumulates phase at a fixed 1.023 MHz. I assumed the NCO could later be aligned with the received code simply by subtracting the measured code phase; but it didn't work. Doppler shift on the 1575.42 MHz carrier is ±5 KHz or ±3 ppm. It also affects the 1.023 Mbps code rate by ±3 chips per second. Fortunately, code Doppler is proportional to carrier Doppler, which is known; and a code phase Doppler correction can be calculated for the elapsed time.
Thin traces are 1-bit, notionally representing ±1. The 2.5 MHz carrier is first despread by mixing with early, late and punctual code. I and Q complex baseband products from the second rank of XOR gate mixers are summed over 10000 samples or 1ms. This low-pass filtering raises SNR 34 dB by reducing noise bandwidth from 2.5 MHz to 1 KHz. Downsampling to 1 KHz necessitates wider onward data paths.
The code tracking loop uses a conventional early-late gate.
Power in the early and late channels is calculated using P = I2 + Q2 which is insensitive to carrier phase θ.
Early and late codes are one chip apart i.e. ½ chip ahead-of and behind punctual.
This diagram helps get the error sense correct:
A Costas Loop is used for carrier tracking and NAV data recovery in the punctual channel. θ is phase error between the received carrier (sans modulation) and local NCO. The I*Q product is actually ½.k2.d2(t).sin(2θ) which is approximately k2θ for small θ, since d(t) is ±1. NAV data, d(t), with the customary 180° phase uncertainty, is taken from the I-arm sign bit. Loop filter F(z) is a discrete-time proportional integral (PI) controller. The same KI and KP are currently used for both loops. Below is a Bode plot of open-loop gain and the step response:
Loop bandwidth is ~ 3 Hz. Noise power in this bandwidth is tiny and the loops can track very weak signals. As there is no software AGC, loop gain and bandwidth vary with signal strength. Because search uses 500 Hz FFT bins, the initial carrier NCO can be up to 250 Hz off-frequency, potentially placing the carrier beyond the loop's grasp. Some GPS receivers have a "pull-in" mode where loop bandwidth only narrows after lock is detected.
The code loop always locks; but the carrier loop occasionally needs help. I noticed carriers close to the original IF centre frequency of 2.5 MHz were most problematic and realised this was due to fractional divider spurs on the NCO. A huge improvement was obtained by shifting the IF frequency up 100 KHz. Being a fractional-N synthesizer, The first local oscillator was easily changed to 1552.82 MHz moving the first and second IF frequencies to 22.6 MHz and 2.6 MHz respectively.
These spectra show the carrier NCO set 50 Hz above IF centres of 2.5 and 2.6 MHz.
The original centre frequency was one quarter of the sampling rate.
The spurs are safely further away when the frequencies are not in a simple ratio.
Despite this improvement, there are still times when the Costas Loop fails to lock, or rather, as a "C" code simulation revealed, acquires a false lock.
With the main goal of solving for user position in sight and not wishing to get bogged-down with this issue,
I opted for a semi-automatic (or semi-manual if your glass is half empty) procedure in which the NCO frequency can be "nudged" by key-presses if the NAV data stream is not recovered right away.
NAV data
The following fragment of NAV data was received at 21:46:45 GMT on Tuesday 1st February 2011:100010110000100101010100110100010100011100011110100111101100100101010101000000000000010111100010110011111111110010110001010111111101010011011010011011110001010110110000110111000001011001001101000100010110111101110010001100001001111001011101111111111111111111101101011111111111011010110101101011100000 100010110000100101010100110100010100011100100000101001100000001101111111000000110111100010001011001100100010011100000100010001100011010011110000001111000011011010101011110111100001010110101011011100001101100111111101101101101001011110110000000011011000111000101010100100001111011000011001111111111000 100010110000100101010100110100010100011100100010101111101000000000000110001111011010001000101010011110010110011100000001000000000011000011011000110010101001100010110001110011010110001001010110000010110010001100000010110001010010111101100011000000000101010011011110011001110010000000100111011111000100 100010110000100101010100110100010100011100100100110001010100011111111010100110011001010110101010011010100110011001010000100110101001100110101001110101010101100110011001100110100110100110011011100110011001001111010101100101011011111111001001111111111111111111111111010110000000000000000000000010001100 100010110000100101010100110100010100011100100110110111011100011100110110001101010101000001000000111111111111111111110111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111010101010101010101010110100 100010110000100101010100110100010100011100101000100110010100100101010101000000000000010111100010110011111111110010110001010111111101010011011010011011110001010110110000110111000001011001001101000100010110111101110010001100001001111001011101111111111111111111101101011111111111011010110101101011100000 100010110000100101010100110100010100011100101010101011111100001101111111000000110111100010001011001100100010011100000100010001100011010011110000001111000011011010101011110111100001010110101011011100001101100111111101101101101001011110110000000011011000111000101010100100001111011000011001111111111000 100010110000100101010100110100010100011100101100101110001100000000000110001111011010001000101010011110010110011100000001000000000011000011011000110010101001100010110001110011010110001001010110000010110010001100000010110001010010111101100011000000000101010011011110011001110010000000100111011111000100 100010110000100101010100110100010100011100101110110011001000011110010101111000011000001100011110011101011100011000111000010001000000011010010111010101110111101111100011100010101101111001111000101010010111000010101010110111100000000010110010000000101010000000101011111100101010101010101010101010111100 100010110000100101010100110100010100011100110000110100111100010000001010101010101010101100101010101010101010101010111100101010101010101010101010111100101010101010101010101010111100101010101010101010101010111100101010101010101010101010111100101010101010101010101010111100101010101010101010101010111100
The above are 2 consecutive frames of 5 subframes each. Subframes are 300-bits long and take 6 seconds to transmit. Column 1 is the preamble 10001011. This appears at the start of every subframe; but can occur anywhere in the data. The 17-bit counter in column 5 is time-of-week (TOW) and resets to zero at midnight Sunday. The 3-bit counter in column 7 is the subframe ID 1 through 5. Subframes 4 and 5 are subcommutated into 25 pages each and a complete data message comprising 25 full frames takes 12.5 minutes to transmit. I am only using data in subframes 1, 2 and 3 at present.
Solving for user position
Every GPS satellite transmits its position and the time. Subtracting time sent from time received and multiplying by the speed of light is how a receiver measures distance between itself and the satellites. Doing so with three satellites would yield three simultaneous equations in three unknowns (user position: x, y, z) if the precise time was available. In practice, receiver clocks are not accurate enough, the exact time is a fourth unknown, four satellites are therefore required and four simultaneous equations must be solved:
An iterative method is used because the equations are non-linear. Using earth's centre (0, 0, 0) and the approximate time as a starting point, the algorithm converges in only five or six iterations. The solution is found even if user clock error is large. The satellites carry atomic clocks; but these too have errors and correction coefficients in subframe 1 must be applied to the time of transmission. Typical adjustments can be hundreds of microseconds.
The uncorrected time of transmission is formed by scaling and adding several counters. Time-of-week (TOW) in seconds since midnight Sunday is sent every subframe. Data edges mark out 20ms intervals within 300-bit subframes. The code repeats 20 times per data bit. Code length is 1023 chips and chip rate is 1.023 Mbps. The code NCO is phase-locked to the signal and accumulated at 10 MHz. All this fixes time of transmission to ± 0.1 µs. Faster NCO clocking could improve on this.
Satellite positions at the corrected transmission time are calculated using ephemeris in subframes 2 and 3. Orbital position at a reference time t_oe (time of ephemeris) is provided along with parameters allowing (x,y,z) position to be calculated up to a few hours before or after. Ephemerides are regularly updated and satellites only transmit their own. Long term orbits of the entire constellation can be predicted less accurately using Almanac data in subframes 4 and 5; however, this is not essential if a fast FFT-based search is used.
Solutions are computed in earth-centred, earth-fixed (ECEF) coordinates. User location is converted to latitude, longitude and altitude with a correction for eccentricity of the earth, which bulges at the equator. The scatter diagrams below illustrate repeatability, the benefit of averaging and the effect of poor satellite choices. Grid squares are 0.001° or ~111 metres on each side. Blue dots mark 1000 fixes. Yellow circles mark the average of each 1000:
(i) North-facing window ledge (ii) Rooftop antenna (iii) East-facing window ledge
The tight cluster (ii) was obtained using satellites in four different quarters of the sky.
Only the rooftop antenna had a clear view in all directions.
But equally good fixes were obtained by averaging, even when half the sky was obscured.
Rooftop fixes also exhibit spreading like (i) and (iii) if the wrong satellites are chosen.
The above solutions were generated without compensating for ionospheric propagation delays using parameters in page 18 of subframe 4 which should be applied because this is a single frequency receiver. Ionospheric refraction increases path lengths between users and satellites. This is probably the largest source of uncorrected error.
I'm grateful to Dan Doberstein for sending me an early draft of his GPS book [2] from which I obtained the solution algorithm. The official US government GPS Interface Specification [3] is an essential reference.
Signal monitor
The above circuit arrangement, mostly implemented in FPGA, de-spreads by taking the product of the 1-bit IF and punctual code, leaving 50 bps data modulation. A small notch due to BPSK carrier suppression can just be seen:
These spectra show the same de-spread transmission at different spans and resolution bandwidths (RBW). Doppler shift was -1.2 KHz. The noise floor is antenna thermal noise amplified and filtered by the IF strip. -3 dB bandwidth looks around 3 MHz, slightly wider than planned. The de-spread carrier is 5 dB above noise at 30 KHz RBW and 25 dB above at 300 Hz RBW. Received signal strength at the antenna can be estimated as -174+1+10*log10(30e3)+5 = -123 dBm.
It still amazes me how well frequency domain information is preserved through hard-limiting! The LVDS transmitter has a constant output current of ~3mA which is ~1mW in 100 ohms. Peak power seen at the SA cannot exceed 0 dBm. Here, we see this available power spread across a range of frequencies. Wideband integrated power spectral density must be ~1mW.
First local oscillator
I've been building experimental fractional-N synthesizers using general-purpose programmable logic for several years:Project Date Technology Frequency (MHz) FracN 2004Altera MAX 7000 CPLD 4.3 Frac2 2005Altera MAX 7000 CPLD 15 - 25 Frac3 2009Xilinx Spartan 3 FPGA38 - 76 Frac42009Xilinx Spartan 3 FPGA38 - 76 Frac5 2010Xilinx Spartan 3 FPGA800 - 1600
The Frac5 board not only synthesizes the 1552.92 MHz first local oscillator but also hosts the GPS receiver's DSP functions in its FPGA. I had no idea Frac5 would be used in a GPS receiver when I originally designed it. The photo below shows how the ROS-1455 VCO output is resistively split between the output SMA and a Hittite HMC363 divide-by-8 prescaler. The 200 MHz divider output is routed (differentially) into the FPGA which phase locks it to a master reference using methods documented in my earlier projects.
High stability and low phase noise are achieved, as can be seen in the VCO output spectra shown below. When this board was originally developed as a dedicated frequency synthesizer, simultaneous toggling of logic on frequencies not harmonically related was avoided to minimise intermodulation spurs. The FPGA was static when clock pulses that toggled phase detector outputs crossed the fabric. No such luxury is practical now the FPGA is almost 100% utilised by a GPS receiver; however, the local oscillator output is still more than good enough:
The Marconi 2383 spectrum analyser's 50 MHz STD OUTPUT is used as the master reference source for Frac5 and all internal GPS receiver clocks. GPS receivers need accuracies better than 1 ppm (parts per million) to measure ±5 KHz Doppler shifts on the 1575.42 MHz L1 carrier. Any frequency uncertainty would necessitate a wider search range.
Source code
Links and resources
- N2YO live tracking of GPS satellites above your horizon
- Celestrak daily GPS ephemeris TLE (two line element) updates
- SPACETRACK Report No. 3 mathematical models for processing orbital elements
- STSPlus orbital tracking software
- How to locate the preamble in NAV data
- PyEphem Python library for astronomical computations
- Doppler.py Python script for predicting GPS satellite Doppler shifts
- www.gps.gov/technical/icwg GPS documentation
References
1. GPS/GLONASS receiver Matjaž Vidmar, S53MV2. PRINCIPLES OF GPS RECEIVERS - A HARDWARE APPROACH by Dan Doberstein
3. IS-GPS-200E GPS Interface Specification
Copyright Š Andrew Holme, 2011.
Comments
Scala on Heroku

The sixth official language on the Heroku polyglot platform is Scala, available in public beta on the Cedar stack starting today.
Scala deftly blends object-oriented programming with functional programming. It offers an approachable syntax for Java and C developers, the power of a functional language like Erlang or Clojure, and the conciseness and programmer-friendliness normally found in scripting languages such as Ruby or Python. It has found traction with big-scale companies like Twitter and Foursquare, plus many others. Perhaps most notably, Scala offers a path forward for Java developers who seek a more modern programming language.
More on those points in a moment. But first, let's see it in action.
Scala on Heroku in Two Minutes
Create a directory. Start with this sourcefile:
src/main/scala/Web.scala
import org.jboss.netty.handler.codec.http.{HttpRequest, HttpResponse}
import com.twitter.finagle.builder.ServerBuilder
import com.twitter.finagle.http.{Http, Response}
import com.twitter.finagle.Service
import com.twitter.util.Future
import java.net.InetSocketAddress
import util.Properties
object Web {
def main(args: Array[String]) {
val port = Properties.envOrElse("PORT", "8080").toInt
println("Starting on port:"+port)
ServerBuilder()
.codec(Http())
.name("hello-server")
.bindTo(new InetSocketAddress(port))
.build(new Hello)
}
}
class Hello extends Service[HttpRequest, HttpResponse] {
def apply(req: HttpRequest): Future[HttpResponse] = {
val response = Response()
response.setStatusCode(200)
response.setContentString("Hello from Scala!")
Future(response)
}
}
Add the following files to declare dependencies and build with sbt, the simple build tool for Scala:
project/build.properties
sbt.version=0.11.0
build.sbt
import com.typesafe.startscript.StartScriptPlugin
seq(StartScriptPlugin.startScriptForClassesSettings: _*)
name := "hello"
version := "1.0"
scalaVersion := "2.8.1"
resolvers += "twitter-repo" at "http://maven.twttr.com"
libraryDependencies ++= Seq("com.twitter" % "finagle-core" % "1.9.0", "com.twitter" % "finagle-http" % "1.9.0")
Declare how the app runs with a start script plugin and Procfile:
project/build.sbt
resolvers += Classpaths.typesafeResolver
addSbtPlugin("com.typesafe.startscript" % "xsbt-start-script-plugin" % "0.3.0")
Procfile
web: target/start Web
Commit to Git:
$ git init
$ git add .
$ git commit -m init
Create an app on the Cedar stack and deploy:
$ heroku create --stack cedar
Creating warm-frost-1289... done, stack is cedar
http://warm-frost-1289.herokuapp.com/ | git@heroku.com:warm-frost-1289.git
Git remote heroku added
$ git push heroku master
Counting objects: 14, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (14/14), 1.51 KiB, done.
Total 14 (delta 1), reused 0 (delta 0)
-----> Heroku receiving push
-----> Scala app detected
-----> Building app with sbt v0.11.0
-----> Running: sbt clean compile stage
Getting net.java.dev.jna jna 3.2.3 ...
...
[success] Total time: 0 s, completed Sep 26, 2011 8:41:10 PM
-----> Discovering process types
Procfile declares types -> web
-----> Compiled slug size is 43.1MB
-----> Launching... done, v3
http://warm-frost-1289.herokuapp.com deployed to Heroku
Then view your app on the web!
$ curl http://warm-frost-1289.herokuapp.com
Hello from Scala!
Dev Center: Getting Started with Scala on Heroku/Cedar
Language and Community
Scala is designed as an evolution of Java that addresses the verbosity of Java syntax and adds many powerful language features such as type inference and functional orientation. Java developers who have made the switch to Scala often say that it brings fun back to developing on the JVM. Boilerplate and ceremony are replaced with elegant constructs, to express intent in fewer lines of code. Developers get all the benefits of the JVM — including the huge ecosystem of libraries and tools, and a robust and performant runtime — with a language tailored to developer happiness and productivity.
Scala is strongly- and statically-typed, like Java (and unlike Erlang and Clojure). Its type inference has much in common with Haskell.
Yet, Scala achieves much of the ease of use of a dynamically-typed language (such as Ruby or Python). Though there are many well-established options for dynamically-typed open source languages, Scala is one of the few with compile-time type safety which is also both practical and pleasant to use. The static vs dynamic typing debate rages on, but if you're in the type-safe camp, Scala is an obvious choice.
Language creator Martin Odersky's academic background shines through in the feel of the language and the community. But the language's design balances academic influence with approachability and pragmatism. The result is that Scala takes many of the best ideas from the computer science research world, and makes them practical in an applied setting.
Members of the Scala community tend to be forward-thinking, expert-level Java programmers; or developers from functional backgrounds (such as Haskell or ML) who see an opportunity to apply the patterns they love in a commercially viable environment.
There is some debate about whether Scala is too hard to learn or too complex. One answer is that the language is still young enough that learning resources aren't yet fully-baked, although Twitter's Scala School is one good resource for beginners. But perhaps Scala is simply a sharper tool than Java: in the hands of experts it's a powerful tool, but copy-paste developers may find themselves with self-inflicted wounds.
Scala Days is the primary Scala conference, although the language is well-represented at cross-community conferences like Strange Loop.
The language community has blossomed, and is now in the process of accumulating more and more mainstream adoption. Community members are enthusiastic about the language's potential, making for an environment that welcomes and encourages newcomers.
Open Source Projects
Open source is thriving in the Scala world. The Lift web framework is a well-known early mover, but the last two years have seen an explosion of new projects showcasing Scala's strengths.
Finagle is a networking library coming out of the Twitter engineering department. It's not a web framework in the sense of Rails or Django, but rather a toolkit for creating network clients and servers. The server builder is in some ways reminiscent of the Node.js stdlib for creating servers, but much more feature-full: fault-tolerance, backpressure (rate-limiting defense against attacks), and service discovery to name a few. The web is increasingly a world of connected services, and Finagle (and Scala) are a natural fit for that new order.
Spark runs on Mesos (a good example of hooking into the existing JVM ecosystem) to do in-memory dataset processing, such as this impressive demo of loading all of Wikipedia into memory for lightning-fast searches. Two other notable projects are Akka (concurrency middleware) and Play! (web framework), which we'll look at shortly.
The Path Forward for Java?
Some Java developers have been envious of modern, agile, web-friendly languages like Ruby or Python — but they don't want to give up type safety, the Java library ecosystem, or the JVM. Leaders in the Java community are aware of this stagnation problem and see alternate JVM languages as the path forward. Scala is the front-runner candidate on this, with support from influential people like Bruce Eckel, Dick Wall and Carl Quinn of the Java Posse, and Bill Venners.
Scala is a natural successor to Java for a few reasons. Its basic syntax is familiar, in contrast with Erlang and Clojure: two other functional, concurrency-focused languages which many developers find inscrutable. Another reason is that Scala's functional and object-oriented mix allows new developers to build programs in an OO model to start with. Over time, they can learn functional techniques and blend them in where appropriate.
Working with Java libraries from Scala is trivial and practical. You can not only call Java libraries from Scala, but go the other way — provide Scala libraries for Java developers to call. Akka is one example of this.
There's obvious overlap here between Scala as a reboot of the Java language and toolchain, and the Play! web framework as a reboot of Java web frameworks. Indeed, these trends are converging, with Play! 2.0 putting Scala front-and-center. The fact that Play! can be used in a natural way from both Java and Scala is another testament to JVM interoperability. Play 2.0 will even use sbt as the builder and have native Akka support.
Typesafe and Akka
Typesafe is a new company emerging as a leader in Scala, with language creator Martin Odersky and Akka framework creator Jonas Bonér as co-founders. Their open-source product is the Typesafe Stack, a commercially-supported distribution of Scala and Akka.
Akka is an event-driven middleware framework with emphasis on concurrency and scale-out. Akka uses the actor model with features such as supervision hierarchies and futures.
The Heroku team worked closely with Typesafe on bringing Scala to our platform. This collaboration produced items like the xsbt-start-script-plugin, and coordination around the release of sbt 0.11.
Havoc Pennington of Typesafe built WebWords, an excellent real-world demonstration of using Akka's concurrency capabilities to scrape and process web pages. Try it out, then dig in on the sourcecode and his epic Dev Center article explaining the app's architecture in detail. Havoc also gave an educational talk at Dreamforce about Akka, Scala, and Play!.
Typesafe: we enjoyed working with you, and look forward to more productive collaboration in the future. Thanks!
Conclusion
Scala's explosive growth over the past two years is great news for both Java developers and for functional programming. Scala on Heroku, combined with powerful toolsets like Finagle and Akka, are a great fit for the emerging future of connected web services.
Further reading:
- Getting Started with Scala on Heroku/Cedar
- Scaling Out with Scala and Akka on Heroku
- Twitter Scala School
- Planet Scala
- Book: Programming in Scala by Martin Odersky, Lex Spoon, and Bill Venners
- Book: Programming Scala by Dean Wampler and Alex Payne
- Video: Scala, Akka, and Play!: An Introduction in the Cloud
Special thanks to Havoc Pennington, Jeff Smick, Steve Jenson, James Ward, Bruce Eckel, and Alex Payne for alpha-testing and help with this post.
Comments
September 28 2011
Getting Started with MMS
Telling someone “You should set up monitoring” is kind of like telling someone “You should exercise 20 minutes three times a week.” Yes, you know you should, but your chair is so comfortable and you haven’t keeled over dead yet.
For years*, 10gen has been planning to do monitoring “right,” making it painless to monitor your database. Today, we released the MongoDB Monitoring Service: MMS.
MMS is free hosted monitoring for MongoDB. I’ve been using it to help out paying customers for a while, so I thought I’d do a quick post on useful stuff I’ve discovered (documentation is… uh… a little light, so far).
So, first: you sign up.

There are two options: register a company and register another account for an existing company. For example, let’s say I wanted to monitor the servers for Snail in a Turtleneck Enterprises. I’ll create a new account and company group. Then Andrew, sys admin of my heart, can create an account with Snail in a Turtleneck Enterprises and have access to all the same monitoring info.

Once you’re registered, you’ll see a page encouraging you to download the MMS agent. Click on the “download the agent” link.

This is a little Python program that collects stats from MongoDB, so you need to have pymongo installed, too. Starting from scratch on Ubuntu, do:
$ # prereqs $ sudo apt-get install python easy_install $ sudo easy_install pymongo $ $ # set up agent $ unzip name-of-agent.zip $ cd name-of-agent $ mkdir logs $ $ # start agent $ nohup python agent.py > logs/agent.log 2>&1 &
Last step! Back to the website: see that “+” button next to the “Hosts” title?

Designed by programmers, for Vulcans
Click on that and type a hostname. If you have a sharded cluster, add a mongos. If you have a replica set, add any member.
Now go have a nice cup of coffee. This is an important part of the process.
When you get back, tada, you’ll have buttloads of graphs. They probably won’t have much on them, since MMS will have been monitoring them for all of a few minutes.
Cool stuff to poke
This is the top bar of buttons:
![]()
Of immediate interest: click “Hosts” to see a list of hosts.
You’ll see hostname, role, and the last time the MMS agent was able to reach this host. Hosts that it hasn’t reached recently will have a red ping time.

Now click on a server’s name to see all of the info about it. Let’s look at a single graph.

You can click & drag to see a smaller bit of time on the graph. See those icons in the top right? Those give you:
That’s the basics. Some other points of interest:
- You can set up alerts by clicking on “Alerts” in the top bar
- “Events” shows you when hosts went down or came up, because primary or secondary, or were upgraded.
- Arbiters don’t have their own chart, since they don’t have data. However, there is an “Arbiters” tab that lists them if you have some.
- The “Last Ping” tab contains all of the info sent by MMS on the last ping, which I find interesting.
- If you are confused, there is an “FAQ” link in the top bar that answers some common questions.
If you have any problems with MMS, there’s a little form at the bottom to let you complain:

This will file a bug report for you. This is a “private” bug tracker, only 10gen and people in your group will be able to see the bugs you file.
* If you ran mongod --help using MongoDB version 1.0.0 or higher, you might have noticed some options that started with --mms. In other words, we’ve been planning this for a little while.
September 27 2011
Stack Overflow Scala Tutorial
Scala is a general purpose programming language principally targeting the Java Virtual Machine. Designed to express common programming patterns in a concise, elegant, and type-safe way, it fuses both imperative and functional programming styles. Its key features are: statically typed; advanced type-system with type inference; function types; pattern-matching; implicit parameters and conversions; operator overloading; full interop with Java

Scala is a general purpose programming language principally targeting the Java Virtual Machine. Designed to express common programming patterns in a concise, elegant, and type-safe way, it fuses both imperative and functional programming styles. Its key features are:
- Statically typed
- Advanced type-system with type inference and declaration-site variance
- Function types (including anonymous) which support closures
- Pattern-matching
- Implicit parameters and conversions which support the typeclass and pimp my library patterns
- Mixin composition
- Full interop with Java
For more information, see the official Scala Introduction.
Stack Overflow Scala Tutorial
- Introduction to Scala
- Variables/values
- Methods
- Equal sign in methods
- Operator notation and the rules
- Leave parentheses in method calls
- Unary operators
- Multiple parameter lists
- Right-associative method names
- Side effect free methods
- Missing parentheses at methods
- Why no i++ in Scala?
- How to mix punctuation with alphanumeric characters in method names?
- Shortcut methods (
+=,-=,*=, ...) - List of "magic method" names (apply, unapply/unapplySeq, update)
- Named arguments / optional parameters
- Type inference in return type
- Literals, statements and blocks
- Difference between block and statement
- Weak conformance
- Symbols
- Difference between braces and parentheses
- Dereference keyword (backtick usage)
- Local definitions
- Loops/recursion
- Data structures / Collections
- Collections design tutorial
- Immutable Collections
- Mutable Collections
- Lazy Collections
- Parallel Collections
- Conversions
- Memory footprint
- How are Scala collections able to return the correct collection type from an operation?
- For-comprehension
- Enumeration
- Pattern-matching
- Explanation
- Value binding (
x @ X) / type binding (x: X) - How to do a multi-match
- Guards
- How to match variables or values?
- How is pattern match implemented under the hood?
- Exhaustive pattern
- Assignment via pattern
- Pattern match in for-expressions
- Ignore cases / no default value
- Conjunction
- Pattern match PartialFunctions
- Match Regex
- Classes, objects and types
- Packages, imports and visibility identifiers
- Imports
- Multiple imports / import renames / hide imports
- Local imports
- Packages
- Visibility
- Explanation
- Private constructors
- Private variables
- Imports
- Inheritance
- Explanation
- Early initializer
- Extractors
- Explanation (Example: conjunctions)
- Infix notation for type parameters (
X[A, B]=>A X B)
- Case classes
- Parameterized types
- How to declare them?
- Upper / lower bounds
- Unify numerics
- How to get around type erasure?
- Abstract Types vs Generics
- Variances
- Traits
- Self references
- Error handling
- Exceptions
- Explanation
- Stand-alone try block
- Ignore an exception
- Option
- Either
- What to use?
- Exceptions
- Type handling
- Annotations
- Functions/Function literals
- Type safety
- Implicits
- Pimp-my-library pattern
- Actors
- Use Java from Scala and vice versa
- XML literals
- Explanation
- Scala Swing
- Explanation
- Examples
- Type Programming
- Functional Scala
Further learning
- Learning Resources
- Operator precedence
- Scala blogs to follow
- Scala style
Comments
September 25 2011
Run a Node PostgreSQL App on Heroku
I really like the pg PostgreSQL library by Brian Carlson, and considering the amount of attention we’ve given to Redis and MongoDB on DailyJS I thought it was time to give relational databases some coverage again.
Heroku is one of many services that supports Node. This tutorial will demonstrate how easy it is to get a simple Express and pg app running.
This tutorial is based on the following documentation:
- Getting Started With Node.js on Heroku/Cedar
- Heroku Database FAQ
- pg’s Documentation
- pg’s Example App
The files for this tutorial can be found here: alexyoung / dailyjs-heroku-postgres.
Getting Started
An account at Heroku is required first. Next, install the Heroku client:
- Heroku for Mac OS X
- Heroku for Windows
- Ubuntu Heroku installation instructions
- For other operating systems, use the Heroku client tarball
Once that’s installed, try typing heroku help in a terminal to see what the command-client client can do. Heroku obviously realised that us developers prefer using the command-line to a GUI — although some basic management features are available through Heroku’s web interface, almost everything is handled from the command-line tool.
Authentication is requried before progressing:
heroku login
I had to tell Heroku about my public SSH key too:
heroku keys:add ~/.ssh/id_rsa.pub
Module Installation
Heroku wisely supports npm, so our app begins with a package.json:
{
"name": "dailyjs-heroku-postgres"
, "version": "0.0.1"
, "dependencies": {
"express": "2.4.5"
, "pg": "0.5.7"
}
}
PostgreSQL Setup
Heroku uses environmental variables to supply database connection parameters. This is simply process.env.DATABASE_URL for PostgreSQL. Connecting to the database is as simple as this:
var pg = require('pg').native
, connectionString = process.env.DATABASE_URL || 'postgres://localhost:5432/dailyjs'
, client
, query;
client = new pg.Client(connectionString);
client.connect();
query = client.query('SELECT * FROM mytable');
query.on('end', function() { client.end(); });
Notice how pg uses events — I’ve called client.end() so this script will exit gracefully when it’s finished. If you’ve got PostgreSQL installed locally you could try experimenting with this script.
Schema
There’s a few ways to change the database schema on Heroku. I’ve made a little schema creation script:
var pg = require('pg').native
, connectionString = process.env.DATABASE_URL || 'postgres://localhost:5432/dailyjs'
, client
, query;
client = new pg.Client(connectionString);
client.connect();
query = client.query('CREATE TABLE visits (date date)');
query.on('end', function() { client.end(); });
I’ll explain how to run this on Heroku later.
Another option would be to use a library like node-migrate by TJ Holowaychuk. I haven’t actually used this before, but it seems like a sensible way to keep local schemas in sync as developers work on a project.
Typing heroku help pg shows the commands available for PostgreSQL, and this includes heroku pg:psql which can be used to open a remote connection to a dedicated database. This won’t be allowed for a shared database, but could be used to modify the schema.
Example App
Now we’ve got a package.json, we just need an app to run. Create a file called web.js that starts like this:
var express = require('express')
, app = express.createServer(express.logger())
, pg = require('pg').native
, connectionString = process.env.DATABASE_URL || 'postgres://localhost:5432/dailyjs'
, start = new Date()
, port = process.env.PORT || 3000
, client;
Notice how I use Heroku’s environmental variable for the database connection string and server port, or defaults for development purposes.
Now we can add the code required to connect to the database:
client = new pg.Client(connectionString);
client.connect();
A single Express route should suffice for this tutorial:
app.get('/', function(req, res) {
var date = new Date();
client.query('INSERT INTO visits(date) VALUES($1)', [date]);
query = client.query('SELECT COUNT(date) AS count FROM visits WHERE date = $1', [date]);
query.on('row', function(result) {
console.log(result);
if (!result) {
return res.send('No data found');
} else {
res.send('Visits today: ' + result.count);
}
});
});
And we better start the app too:
app.listen(port, function() {
console.log('Listening on:', port);
});
Procfile
The last thing we need is a file that tells Heroku what our main script is called. Create a file called Procfile:
web: node web.js
Deploying
Heroku uses Git for deployment, so set up a repo:
git init
git add .
git commit -m 'First commit'
Then run this command which creates a remote app on the service with a random name:
heroku create --stack cedar
It’ll give you the URL, but your app isn’t quite ready yet.
Now push the repo to make the magic happen:
git push heroku master
And tell Heroku you want to use a database:
heroku addons:add shared-database
And finally… run the schema creation script:
heroku run node schema.js
Hopefully you now have a little Node and PostgreSQL app running on Heroku!
If anything went wrong, Heroku’s documentation is excellent, and you can download my sample source here: alexyoung / dailyjs-heroku-postgres.
September 20 2011
Merging multiple RSS Feeds into one
Merging multiple RSS Feeds into one
RSS (Really Simple Syndication) is a web feed format that is used to publish frequently updated work on websites.
If we need feeds from multiple locations, we can easliy merge it into one. The best tool with the perfect UI that I would recommend is Yahoo Pipes. Using Yahoo Pipes we can merge multiple RSS feeds and it has many customization options. You can create feeds as per your requirement. Yahoo Pipes filtering option gives you the flexibility to merge feeds as the way you want.
Following are the steps to merge multiple RSS feeds using Yahoo Pipes:
Step 1 : Visit http://pipes.yahoo.com/pipes/
Step 2 : Log in using your yahoo login details.
Step 3 : Click on create a pipe.
Step 4 : Drag and drop Fetch feed from left and enter the feed URL.
Step 5 : Repeat step 4 for multiple feeds.
Likewise you can add multiple feed as per your needs.
Step 6 : You can also filter the feed data. Drag and drop Filter from Operators-> filter and set your filtering parameters. You can also add multiple operators as you need. For example:
You get your customized output in Pipe Output.
Step 7 : Save the pipe.
How to use your customized feed :
1. Go to my pipes, here you can see list of pipes you have created, click on one of the pipe:
2. Click on Get RSS and your RSS link is ready.
You can embed this feed on your website and enjoy multiple feeds at one time. :)
Comments
September 17 2011
September 16 2011
Announcing Joyent Cloud (joyentcloud.com)
San Francisco — September 15, 2011 —Joyent, the global provider of cloud computing software and services, today announced the launch of a new public cloud computing platform that delivers a powerful combination of superior application speed, 100% data resiliency, and granular analytics. The new Joyent Cloud enables businesses and developers to run applications faster and with greater peace of mind for less money than on any other cloud.
The new Joyent Cloud runs on a completely revamped Joyent infrastructure management platform, SmartDataCenter 6. This new platform adds unique analytics, management, security, and performance capabilities to the Joyent Cloud. It provides customers with the most complete environment for running fast, instantly scalable applications without paying premium prices or additional service fees.
In addition to the speed, data integrity, uptime, and world-class collaborative support that Joyent has always offered, Joyent Cloud now delivers the following benefits to enterprises and developers:
• The best visibility into one’s cloud infrastructure: Powerful cloud analytics from Joyent afford customers complete transparency into sources of latency and root cause throughout the compute stack from kernel to application. Coupled with New Relic application analytics and Zeus load balancing software, customers have the tools to locate and fix problems before they cause slowdowns or downtime.
• Safe storage, resilient data, and speed. Lighting-fast Linux and Windows SmartMachines boot in seconds and provide fully distributed storage, superior application and processing speeds, data fidelity and security not available to those operating systems on other clouds. Joyent’s SmartOS SmartMachine offers the same benefits plus even more granular analytics and enhanced vertical scaling.
• Lower compute costs. Joyent’s SmartOS SmartMachines use fewer servers and lower compute costs by delivering up to 14x faster speeds than comparable Amazon EC2 machine instances and providing instant CPU bursting and intelligent data caching.
• Price predictability and simplicity. A new, simple utility pricing model that gives customers a clear view of their costs and simplifies resource planning while preserving flexibility.
Joyent Cloud customers include LinkedIn, Kabam, StackMob, and Gilt Groupe, among numerous leading companies in ecommerce, mobile, online games and other markets. Customers have enjoyed years of superior application performance on Joyent Cloud compared to less performant clouds and 99.99% uptime, with annual downtime measured in seconds, not hours or days.
“Joyent Cloud provides customers with peace of mind that their infrastructure will just work, just scale, and be fully visible and transparent at all times,” said Steve Tuck, GM of Joyent Cloud. “We eliminate the compromises inherent in running on other clouds while delivering exceptional performance and speed at a competitive price.”
Joyent Cloud delivers CPU bursting capability of up to 400%, which allows applications to easily absorb abrupt usage spikes without spinning up new instances or impacting end user experience or application performance. With Joyent Cloud’s new public APIs, operators and developers can easily configure and change their Joyent Cloud SmartMachine with their own custom script packages or via third-party cloud management platforms. And Joyent Cloud offers two free analytics packages – Joyent Cloud Analytics and New Relic that for the first time give businesses complete, real-time, visibility into latency issues anywhere from their virtual CPU up through the application layer across multiple Joyent SmartMachines.
In addition, Joyent Cloud is introducing a Node.js Joyent SmartMachine for customers ready to deploy commercial Node.js-based applications. Joyent also hosts the Node.js service, No.de, a Platform as a Service offering for developers wishing to create data-intensive, real-time (DIRTy) apps. Joyent Cloud also offers customers access to the services of a rich ecosystem of partners, including StackMob, provider of a powerful backend development and deployment platform for mobile applications, AppFog, the cloud-based hosting service for customers’ favorite web application stack, and Circonis, the organization-wide monitoring company.
To learn more about JoyentCloud, please visit http://joyentcloud.com.
About Joyent
Joyent is a global cloud computing software and service provider that offers an integrated technology suite designed for service providers, enterprises, and developers. JoyentCloud.com delivers public cloud services to some of the most innovative companies in the world, including LinkedIn, Gilt Groupe and Kabam. Joyent licenses its cloud software to service providers that include Dell, FirstServer, XYBase, ClusterTech and Uniserve. Joyent offerings also include Platform-as-a-Service based on Node.js, the open source server-side JavaScript development environment that has been embraced by enterprises around the world, such as Microsoft. Joyent is also the key contributor to and sponsor of Joyent SmartOS, an open source project dedicated to the complete, modern operating system. A global ecosystem of leading technology partners assists Joyent in enabling customers to leverage the performance, scalability, reliability and security inherent in the company’s cloud solutions. For more information, visit http://www.joyent.com.
Joyent Media Contact
Scott VanSickle
The Hoffman Agency
+1-408-835-7547
svansickle@hoffman.com
Comments
September 15 2011
Details on MSN’s XMPP server
Mickaël posted yesterday On MSN / Live Messenger adopting XMPP. Today, we’ll dig a little more in technical details.
Here are some technical details and some explainations, as well as some questions.
Microsoft’s public XMPP server
Microsoft’s public XMPP server is located at: xmpp:messenger.live.com. You can double check on IMtrends:
http://www.imtrends.com/do/search_domain_simple?domain=messenger.live.com&x=19&y=7
Microsoft had been testing xmpp:beta.xmpp.messenger.live.com since a few months, but now it is redirecting to xmpp:messenger.live.com.
New server?
IMtrends test results say:
Software IMTrends couldn't determine the server running behind messenger.live.com.
This makes sense, since it may be a completely new server.
Server features
Aditionally:
Features We couldn't obtain the list of features
This says that the discovery of services offered by the server and/or the stream features are not available: this means that the most necessary, important and basic feature is not implemented, or implemented but not the open standard way. Maybe they are working on it though. Maybe the official XMPP/MSN client knows the server features.
C2S, but no S2S
C2S stands for client-to-server and S2S stands for server-to-server, which are meaningfull: these describe the connections between clients and servers.
On this matter, IMtrends says:
OK:
- DNS Client-To-Server record
- Client-To-Server Stream
Not OK:
- DNS Server-To-Server record
- Server-To-Server Stream
What does this mean?
This means:
- XMPP clients may be able to connect to this XMPP service
- XMPP servers are not allowed to connect to this XMPP server: Microsoft’s XMPP server does not federate with Google’s, like Facebook’s.
- As a consequence, we still need gateways (“transports”) to MSN service, with MSNP or XMPP protocols, in order to aggregate MSN contacts in an XMPP client.
Questions:
- Will Microsoft’s XMPP server federate with Google’s and the rest of the world?
- Microsoft has invested in Facebook: will Facebook’s XMPP server federate with Microsoft’s, Google’s and the rest of the world?
- Microsoft has bought Skype: will Skype (which already offers XMPP on their client) offer an XMPP server? Will it federate?
JID, Jabber ID, XMPP addresses
Justin Karneges of Psi and Livefyre fame, brought our attention to JID and authentication.
JIDs are in the form: [identifier]@messenger.live.com.
Which means:
- Users will have long JIDs, instead of the short
[username]@live.com: it would have been simpler to provide the JID as the email address, like it is the case on Gmail/Gtalk and TextOne - I hope
[identifier]is the username, or something human-readable…
Authentication mechanism
The client-to-server authentication used is SASL, with a specific and prioprietary mechanism called X_MESSENGER_OAUTH2. According to Thijs Alkemade (Adium), it is documented, and “extremely similar to Facebook’s OAuth2 mechanism”.
Which means that all current XMMP clients are NOT able to connect to messenger.live.com. This new authentication schema will need to be developped and tested.
Summary
Here is a small summary:
- Microsoft only offers a client interface to their MSN chat, much like Facebook: they both keep their internal and proprietary chat system
- Current standard XMPP clients can not connect to Microsoft’s XMPP server, since it has a specific and proprietary authentication schema, unlike Facebook
- Microsoft’s XMPP server does not talk to any public and federated XMPP server.
Perspective
This is a big step, and quite a surprise at internet scale (even if some were aware of their beta server). Indeed Microsoft has defended over time their proprietary protocol MSNP (and it’s mobile version) by changing small bits of their protocol in order to prevent third party clients to connect to their service. A big step, but still a long way to go until full interop and federation with the full XMPP network, including Gtalk, Facebook, and Skype (soon AIM?). ICQ, Yahoo! and QQ are still lagging behind.
Now, given Microsoft’s habits to cheat on interop, and “embrace and extend” the open standards, it is needed, not only to run deeper and more strict and exhaustive interop tests, but also run ACID-like tests over XMPP.
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...




