Module bhreader

Using bhreader module

Note

It is rarely necessary to call bhreader directly, instead call loader.loadfile_bh()

This module contains functions to load and decode files from Becker & Hickl (B&H) hardware.

The main, highest level function of the module is load_spc(). This function will automatically detect the type of card, and read the files appropriately.

load_spc() returns a dictionary of with 2 keys: ‘meta’, and ‘photon_data’:

• ‘meta’: the dictionary loaded by load_set()

• ‘photon_data’: dictionary of the data loaded from the .spc file, all values are already converted into form acceptable for photon-hdf5

  • ‘timestamps’: numpy array of timestamps of all photons/markers

  • ‘detectors’: numpy array of the detector of each photon/marker (markers already reassigned unique index)

  • ‘nanotimes’: nanotime of each photon (ADC automatically converted to nanotime)

  • ‘timestamps_unit’ : the timestamps unit of macrotimes

  • ‘marker_ids’ : array of unuiqe indexes cooresponding to detector indexes that should be read as markers

:func`load_spc` loads just the .set file which contains metadata on the .spc file. It returns a dictionary with the following keys:

• ‘identification’: a dictionary of key value pairs that are both strings equivalent to the IDENTIFICATION portiion of the .set file

• ‘setup’: a dictionary of key (string) value (any) pairs contained in the SETUP section of the .set file. This contains metadata on the acqusition settings

• ‘header’: 1 element numpy array with a dtype representing the .set header structure (see following explanation) this most importantly contains a code indicating the SPC card used (see bhformats )

Code examples

load_spc() is flexible, assuming the .set and .spc files have the same name other than the extension, it is easiest, and best to specify the .spc file

import phconvert as phc
data = phc.bhreader.load_spc('sample.spc')

however, the result will be the same as

data = phc.bhreader.load_spc(setfile='sample.set')

If for some reason, the .set and .spc files have different names, it is necessary to specify both

data = phc.bhreader.load_spc(spcfile='sample.spc', setfile='Sample1.set')

Warning

SPCM sofware never saves the .set and .spc files with different names. Therefore, this situation indicates that the files were manually renamed. Renaming the two files differently is therefore highly inadvisable.

Future proofing

With the latest version of phconvert, detection of the spc card and therefore record format has become automatic.

This does come with a downside: if B&H comes out with a new card, it will be unrecognizable by phconvert versions published before the new card.

However, in anticipation of this, a work-around is provided. (As of writing (July 20205), there should be no cards for which this is necessary)

Assuming B&H has not come up with another format (i.e. your card is new, but uses an existing record format), load_spc() can be instructed to read the .spc file according the the specified format.

This is done by specifying the ‘SPC_type’ argument of load_spc() as a string, see the first line of the subsections of bhformats to get the argument to provide.

For example

data = phc.bhreader.load_spc('sample.spc', SPC_type='SPC-1XX')

Becker & Hickl SPC formats

Becker & Hickl documentation for their file format is scattered across the TCSPC handbook and in the SPC_data_file_structure.h file which can be located in the SPCM sofware directory within the program files directory after installation of the B&H SPCM sofware.

The format of the .set file is consistent across all versions and TCSPC cards. The .spc file however has several potential layout depending on the B&H card used to acquire the data.

The .set file starts with binary data according to a custom structure defined in SPC_data_file_structure.h, most notably the first 2 bytes contain information on the version of the sofware, and TCSPC card used. Next are 2 ascii formated sections, IDENTIFICATION and SETUP. IDENTIFICATION contains general user specified data, while SETUP contains specific information on settings of the TCSPC acquisition.

Depending on the B&H TCSPC card, the data will be stored according to one of the following formats. In all cases, the 1st 48 or 32 bits are a header that defines the macrotime clock (timestamps_unit), and number of routing bits. After the header, all subsequent data are records of single photon arrivals, each record is always the same number of bytes as the header.

SPC-1XX and SPC-8XX format

load_set() can be forced to read the .spc in this format by setting SPC_type='SPC-1XX'

The majority of modern B&H cards use this format, whic relies on 32 bit (4 byte) records. The structure of these records is as follows

Bit	31	30	29	28	27-16	15-12	11-0
name	Invalid	Macrotime overflow	Gap	marker	ADC	Routing	Macrotime
type	bool	bool	bool	bool	int 12 bit	int 4 bit	int 12 bit

Where:

• invalid: boolean indicating if record can be considered a photon, if 1, then the record either arose from an error, or if macrotime overflow is also 1, then exclusively as a macrotime overflow (see below)

• Macrotime overflow: since only 12 bits are allocated to the macrotime every \(2^{12} = 4096\) macrotime unit, it is necessar to add an overflow event, which adds an additional 4096 to the value of the macrotime. So, for any given record, the number of overflow events in the records up to the given record must be summed, and \(4096 * no.\:of\:overflows\) must be added to the macrotime clock

• Gap: this indicates that the file buffer overflowed, and therefore there may be photons missed in the recording. This is rare and indicates potentially corupted data

• marker indicates if the record is a marker photon, ie for FLIM, if the bit is 1, then the record must be read differently (see below)

• 0: this bit is always zero and serve no purpose

• ADC: this represents the nanotime, but as the cards record in stop-start mode the nanotime = 4096 - ADC

• Routing: the index of the detector from the router, ie this indicates which detector the photon arrived at

• Macrotime: the time since the last overflow event, in macrotime units

The first record in any .spc file follows the following format

Bit	31	30-78	26	25	24	23-0
name	1	unused	raw	markers	reserved	macrotime unit
type	–	–	bool	bool	–	int 24 bit

• macrotime unit is in 0.1 nanoseconds

• raw indicates if data was recorded in diatnostic mode

• markers indicated if markers are used in the photon stream

Marker records (bit 28 = 1) change certain field

• routing becomes the marker type

• ADC is irrelevant, these values are ignored

• bit 31 (Invalid) is set to 1

SPC-QC-Formats

Bit	31	30-27	26	25	24	23	22	21-0
	1	number of routing bits	raw	markers	femto	6 channel	unused	macrotime unit

SPC-QC-X06/X08 format

load_spc() can be forced to read the .spc file in this format by specifying SPC_type='SPC-QC-X04'

QC-004 type cards store the data in the following 32-bit record format

Photon records use the following format

Bit	31 -30	29-28	27-16	15-12	11-0
name	record type	channel	nanotime	routing	macrotime
type	int 2 bit	int 2 bit	int 12 bit	int 4 bit	int 12 bit

The record codes are the following

• bit 31 = 0, bit 30 = 0: normal photon record

• bit 31 = 1, bit 30 = 0: macrotime overflow, all other bits 0 by definition

• bit 31 - 0, bit 30 = 1: marker, channel and nanotime bits all 0 by definition

• bit 31 = 1, bit 30 = 1: Gap, which occurs immediately before a file in file out overflow, meaning potential missing data, read remaining bytes as normal photon

For normal photon records

• channel indicates which input channel the event occured with

• routing indicates the routing signal for the photon

• To determine detector, both channel and routing must be considered

• unlike most other B&H record formats, the nanotime is not inverted, ie 0x000 = 0 ns

For marker records:

• routing number indicates the marker type

The first “header” records is in the following format

SPC-QC-X06/X08 format

load_spc() can be forced to read the .spc file in this format by specifying SPC_type='SPC-QC-X06'

QC-X0(>4) cards use a 32-bit record format similar to that of QC-X04 cards, but with some key changes due to having more channels than can be indexed with the 2 bits allocated in the QC-0X04 format.

Bit	31	30-28	27-16	15-12	11-0
name	Special	record/channel	nanotime	routing	macrotime
type	bool	int 3 bit	int 12 bit	int 4 bit	int 12 bit

As in QC-004, channel is the physical channel, and routing the signal from the router. Therefore to determine the detctor, both must be taken into account.

If Special (bit 31) = 1, then the bits 30-2 are used to determine the non-photon type of record.

Bit	31	30	29	28	27-16	15-12	11-0
photon	0	ch[2:0]			nanotime	routing	macrotime
macrotime overflow	1	0	0	0	0x000	0x0	0x000
marker	1	0	1	0	0x000	marker	macrotime
Gap	1	1	ch[1:0]		nanotime	routing	macrotime

The first record follows the same format as QC-X04 cards

SPC-6XX formats

Older SPC-6XX cards can use either a 48 bit or 32 bit size records. Since there are 2 formats for these cards, the format must be infered. This is normally done automatically, based on the initial bits of .spc file.

load_spc() can be forced to ignore the .set file, and perform the inference as an SPC-6XX card by specifying SPC_type='SPC-6XX'

The 48 bit format has the following bit assigments

load_spc() can be forced to read .spc files in this fomat by specifying SPC_type='SPC-6XX-48bit'

Bit	47-32	31-24	23-16	15	14	13	12	11-0
name	macrotime[15:0]	routing	marcrotime[23:12]	0	Gap	macrotime overflow	Invalid	ADC
type	int 24 bit[15:0]	int 8 bit	int 24 bit[23:12]	–	bool	bool	bool	int 12 bit

The 32 bit fomat has the following bit assigments

load_spc() can be forced to read .spc files in this fomat by specifying SPC_type='SPC-6XX-48bit'

Bit	31	30	29	28	27	26-8	7-0
name	Invalid	macrotime overflow	GAP	0	routing	macrotime	ADC
type	bool	bool	bool	–	int 3 bit	int 17 bit	int 8 bit

Where

• invalid: boolean indicating if record can be considered a photon, if 1, then the record either arose from an error, or if macrotime overflow is also 1, then exclusively as a macrotime overflow (see below)

• Routing: the index of the detector from the router, ie this indicates which detector the photon arrived at

• Gap: this indicates that the file buffer overflowed, and therefore there may be photons missed in the recording. This is rare and indicates potentially corupted data

• Macrotime overflow: due to the limited number of bit available to the macrotime, every \(2^{num\:bits}\) macrotime unit, it is necessar to add an overflow event, which adds an additional \(2^{num\:bits}\) to the value of the macrotime. So, for any given record, the number of overflow events in the records up to the given record must be summed, and \(2^{num\:bits} * no.\:of\:overflows\) must be added to the macrotime clock

• Macrotime: the time since the last overflow event, in macrotime units

• ADC: this represents the nanotime, but as the cards record in stop-start mode the nanotime = \(max(ADC) - ADC\)

Note

For the 48 bit format, the macrotime is stored in 2 blocks, which must be combined, the 47-32 block stores the smaller values, but otherwise all integer values stored in little endian format

List of functions

High-level functions to load and decode Becker & Hickl SPC/SET pair of files:

phconvert.bhreader.load_set(setfile: str | bytes | PathLike) → dict

Read set file into dictionary.

Parameters:

setfile (str, bytes, os.PathLike) – Path to set file of data.

Returns:

meta –

A dictionary of the metadata with 3 keys:

• ’header’: the header of the set file, represented as single element array with custom numpy.dtype

• ’identification’: a dictionary of the key:value pairs represented in the IDENTIFICATION section of the set file

• ’setup’: a dictionary of the key:value pairs represented in the SETUP section of the set file (formerly sys_params)

Return type:

dict

phconvert.bhreader.load_spc(spcfile: str | bytes | PathLike | None = None, setfile: str | bytes | PathLike | None = None, SPC_type: str = 'auto') → dict

Read data from .spc/.set file pair. Must specify either spcfile or setfile, and function will infer the name of the other.

If you are using an SPC-card that is newer than the current publication of phconvert, you will need to specify SPC_type, which specifies the format of the data in the .spc file. Options are:

• ‘SPC-1XX’ for SPC-1XX and SPC-8XX cards (this is the most common format)

• ‘SPC-6XX’ for SPC-6XX cards, and will infer if saved in 32 or 48 bit formats

• ‘SPC-6XX-48bit’ for SPC-6XX cards with 12 bit TCSPC resolution (most common for these cards)

• ‘SPC-6XX-32bit’for SPC-6XX cards with 8 bit TCSPC resolution (most common for these cards)

• ‘QC-X04’ for SPC-QC-X04 cards

• ‘QC-X06’for SPC-QC-X06 cards

Parameters:

• spcfile (str, bytes, os.PathLike, conditionally optional) – Path to the .spc file, must specify if setfile is not specified

• setfile (str, bytes, os.PathLike, conditionally optional) – Path to the .set file, must specify if spcfile is not specified

• SPC_type (str) – Used if your card is newer than phconvert, otherwise leave as ‘auto’ See above the options for cards.

Returns:

data – Dictionary with 2 keys containing the following data

• ’meta’ the metadata from the .set file, with header, identity and setup sub-keys (see load_set() for more details)

• ’photon_data’ dictionary containing arrays of photon data for timestamps, detectors, and nanotimes, as well as an array of marker_ids indicating the id of detectors that are markers, and timestamps_unit, a float indicating the clock rate of timestamps

Return type:

dict

Low-level functions

These functions are the building blocks for decoding Becker & Hickl files:

phconvert.bhreader.bh_decode(s)

Replace code strings from .SET files with human readable label strings.

phconvert.bhreader.bh_print_sys_params(sys_params)

Print a summary of the Becker & Hickl system parameters (.SET file).