This class is used to cache DAP2 response objects. More...

#include <BESStoredDapResultCache.h>

Inheritance diagram for BESStoredDapResultCache:

Collaboration diagram for BESStoredDapResultCache:

Public Member Functions
virtual bool	cache_too_big (unsigned long long current_size) const
	look at the cache size; is it too large? Look at the cache size and see if it is too big. More...

virtual bool	create_and_lock (const string &target, int &fd)
	Create a file in the cache and lock it for write access. More...

virtual void	dump (ostream &strm) const
	dumps information about this object More...

virtual void	exclusive_to_shared_lock (int fd)
	Transfer from an exclusive lock to a shared lock. More...

virtual string	get_cache_file_name (const string &src, bool mangle=false)
	Build the name of file that will holds the uncompressed data from 'src' in the cache. More...

virtual unsigned long long	get_cache_size ()
	Get the cache size. More...

libdap::DDS *	get_cached_dap2_data_ddx (const std::string &cache_file_name, libdap::BaseTypeFactory *factory, const std::string &dataset)
	Read data from cache. More...

libdap::DMR *	get_cached_dap4_data (const string &cache_file_name, libdap::D4BaseTypeFactory *factory, const string &filename)
	Read data from cache. More...

virtual bool	get_read_lock (const string &target, int &fd)
	Get a read-only lock on the file if it exists. More...

const string	getCacheDirectory ()

const string	getCacheFilePrefix ()

virtual bool	getExclusiveLock (string file_name, int &ref_fd)
	Get an exclusive read/write lock on an existing file. More...

virtual void	lock_cache_read ()
	Get a shared lock on the 'cache info' file. More...

virtual void	lock_cache_write ()
	Get an exclusive lock on the 'cache info' file. More...

virtual void	purge_file (const string &file)
	Purge a single file from the cache. More...

virtual string	store_dap2_result (libdap::DDS &dds, const std::string &constraint, BESDapResponseBuilder rb, libdap::ConstraintEvaluator eval)

virtual string	store_dap4_result (libdap::DMR &dmr, const string &constraint, BESDapResponseBuilder *rb)

virtual void	unlock_and_close (const string &target)
	Unlock the named file. More...

virtual void	unlock_cache ()
	Unlock the cache info file. More...

virtual void	update_and_purge (const string &new_file)
	Purge files from the cache. More...

virtual unsigned long long	update_cache_info (const string &target)
	Update the cache info file to include 'target'. More...

virtual	~BESStoredDapResultCache ()

Static Public Member Functions
static string	assemblePath (const string &firstPart, const string &secondPart, bool addLeadingSlash=false)

static BESStoredDapResultCache *	get_instance (const string &bes_catalog_root_dir, const string &stored_results_subdir, const string &prefix, unsigned long long size)

static BESStoredDapResultCache *	get_instance ()
	Get the default instance of the BESStoredDapResultCache object. More...

Static Public Attributes
static const string	PREFIX_KEY = "DAP.StoredResultsCache.prefix"

static const string	SIZE_KEY = "DAP.StoredResultsCache.size"

static const string	SUBDIR_KEY = "DAP.StoredResultsCache.subdir"

Protected Member Functions
	BESStoredDapResultCache (const string &data_root_dir, const string &stored_results_subdir, const string &prefix, unsigned long long size)

void	initialize (const string &cache_dir, const string &prefix, unsigned long long size)

Friends
class	ResponseBuilderTest

class	StoredDap2ResultTest

class	StoredDap4ResultTest

Detailed Description

This class is used to cache DAP2 response objects.

Author: jhrg 5/3/13

Definition at line 52 of file BESStoredDapResultCache.h.

Constructor & Destructor Documentation

BESStoredDapResultCache::BESStoredDapResultCache	(	const string &	data_root_dir,
		const string &	stored_results_subdir,
		const string &	prefix,
		unsigned long long	size
	)

protected

Definition at line 224 of file BESStoredDapResultCache.cc.

virtual BESStoredDapResultCache::~BESStoredDapResultCache ( )

inlinevirtual

Definition at line 99 of file BESStoredDapResultCache.h.

Member Function Documentation

string BESStoredDapResultCache::assemblePath	(	const string &	firstPart,
		const string &	secondPart,
		bool	addLeadingSlash = `false`
	)

static

Definition at line 161 of file BESStoredDapResultCache.cc.

Referenced by BESDapResponseBuilder::store_dap2_result(), and BESDapResponseBuilder::store_dap4_result().

bool BESFileLockingCache::cache_too_big ( unsigned long long current_size ) const

virtualinherited

look at the cache size; is it too large? Look at the cache size and see if it is too big.

Returns: True if the size is too big, false otherwise.

Definition at line 663 of file BESFileLockingCache.cc.

Referenced by BESDapResponseCache::cache_dataset(), gateway::RemoteHttpResource::retrieveResource(), BESUncompressManager3::uncompress(), BESFileLockingCache::update_and_purge(), and BESH4Cache::write_cached_data().

bool BESFileLockingCache::create_and_lock	(	const string &	target,
		int &	fd
	)

virtualinherited

Create a file in the cache and lock it for write access.

If the file does not exist, make it, open it for read-write access and get an exclusive lock on it. The locking operation blocks, although that should never happen.

Parameters

target	The name of the file to make/open/lock
fd	Value-result param that holds the file descriptor of the opened file

Returns: True if the operation was successful, false otherwise. This method will return false if the file already existed (the file won't be locked and the descriptor reference is undefined - but likely -1).

Exceptions

BESBESInternalErroror if any error except EEXIST is returned by open(2) or if fcntl(2) returns an error.

Definition at line 476 of file BESFileLockingCache.cc.

References BESDEBUG, BESFileLockingCache::lock_cache_write(), and BESFileLockingCache::unlock_cache().

Referenced by BESDapResponseCache::cache_dataset(), gateway::RemoteHttpResource::retrieveResource(), BESUncompressManager3::uncompress(), and BESH4Cache::write_cached_data().

Here is the call graph for this function:

void BESFileLockingCache::dump ( ostream & strm ) const

virtualinherited

dumps information about this object

Displays the pointer value of this instance along with information about this cache.

Parameters

strm	C++ i/o stream to dump the information to

Implements BESObj.

Reimplemented in BESCache3.

Definition at line 901 of file BESFileLockingCache.cc.

References BESIndent::Indent(), BESIndent::LMarg(), and BESIndent::UnIndent().

Here is the call graph for this function:

void BESFileLockingCache::exclusive_to_shared_lock ( int fd )

virtualinherited

Transfer from an exclusive lock to a shared lock.

If the file has an exclusive write lock on it, change that to a shared read lock. This is an atomic operation. If the call to fcntl(2) is protected by locking the cache, a dead lock will result given typical use of this class. This method exists to help with the situation where one process has the cache locked and is blocking on a shared read lock for a file that a second process has locked exclusively (for writing). By changing the exclusive lock to a shared lock, the first process can get its shared lock and then release the cache.

Parameters

fd	The file descriptor that is exclusively locked and which, on exit, will have a shared lock.

Definition at line 506 of file BESFileLockingCache.cc.

Referenced by BESDapResponseCache::cache_dataset(), gateway::RemoteHttpResource::retrieveResource(), and BESUncompressManager3::uncompress().

string BESStoredDapResultCache::get_cache_file_name	(	const string &	local_id,
		bool	mangle = `false`
	)

virtual

Build the name of file that will holds the uncompressed data from 'src' in the cache.

Note: This method overrides the parent methid and DOES NOT support mangling the name. The "mangle" parameter will be ignored.

Parameters

src	The source name to cache
mangle	if True, assume the name is a file pathname and mangle it. If false, do not mangle the name (assume the caller has sent a suitable string) but do turn the string into a pathname located in the cache directory with the cache prefix. the 'mangle' param is true by default.

Reimplemented from BESFileLockingCache.

Definition at line 861 of file BESStoredDapResultCache.cc.

References BESDEBUG.

unsigned long long BESFileLockingCache::get_cache_size ( )

virtualinherited

Get the cache size.

Read the size information from the cache info file and return it. This methods locks the cache.

Returns: The size of the cache.

Definition at line 675 of file BESFileLockingCache.cc.

References BESFileLockingCache::lock_cache_read(), and BESFileLockingCache::unlock_cache().

Referenced by BESFileLockingCache::purge_file().

Here is the call graph for this function:

DDS * BESStoredDapResultCache::get_cached_dap2_data_ddx	(	const std::string &	cache_file_name,
		libdap::BaseTypeFactory *	factory,
		const std::string &	dataset
	)

Read data from cache.

Allocates a new DDS using the given factory.

Definition at line 499 of file BESStoredDapResultCache.cc.

References BESDEBUG.

DMR * BESStoredDapResultCache::get_cached_dap4_data	(	const string &	cache_file_name,
		libdap::D4BaseTypeFactory *	factory,
		const string &	filename
	)

Read data from cache.

Allocates a new DDS using the given factory. If the file does not exists this will return null (0).

Definition at line 541 of file BESStoredDapResultCache.cc.

References BESDEBUG.

BESStoredDapResultCache * BESStoredDapResultCache::get_instance	(	const string &	bes_catalog_root_dir,
		const string &	stored_results_subdir,
		const string &	prefix,
		unsigned long long	size
	)

static

Definition at line 235 of file BESStoredDapResultCache.cc.

References BESDEBUG, and BESError::get_message().

Here is the call graph for this function:

BESStoredDapResultCache * BESStoredDapResultCache::get_instance ( )

static

Get the default instance of the BESStoredDapResultCache object.

This will read "TheBESKeys" looking for the values of SUBDIR_KEY, PREFIX_KEY, an SIZE_KEY to initialize the cache.

Definition at line 254 of file BESStoredDapResultCache.cc.

References BESDEBUG, and BESError::get_message().

Referenced by BESDapResponseBuilder::store_dap2_result(), and BESDapResponseBuilder::store_dap4_result().

Here is the call graph for this function:

bool BESFileLockingCache::get_read_lock	(	const string &	target,
		int &	fd
	)

virtualinherited

Get a read-only lock on the file if it exists.

Try to get a read-only lock on the file, blocking until we can get it. If the file does not exist, return false.

Note: If this code returns false, that means the file did not exist in the cache at the time of the test. by the time the caller gets the result, the file may have been added to the cache by another process.

Parameters

src	src file that will be cached eventually
target	a value-result parameter set to the resulting cached file

Returns: true if the file is in the cache and has been locked, false if the file is/was not in the cache.

Exceptions

Error if the attempt to get the (shared) lock failed for any reason other than that the file does/did not exist.

Definition at line 448 of file BESFileLockingCache.cc.

References BESDEBUG, BESFileLockingCache::lock_cache_read(), and BESFileLockingCache::unlock_cache().

Referenced by BESDapResponseCache::cache_dataset(), BESH4Cache::get_data_from_cache(), gateway::RemoteHttpResource::retrieveResource(), and BESUncompressManager3::uncompress().

Here is the call graph for this function:

const string BESFileLockingCache::getCacheDirectory ( )

inherited

Definition at line 888 of file BESFileLockingCache.cc.

const string BESFileLockingCache::getCacheFilePrefix ( )

inherited

Definition at line 884 of file BESFileLockingCache.cc.

bool BESFileLockingCache::getExclusiveLock	(	string	file_name,
		int &	ref_fd
	)

virtualinherited

Get an exclusive read/write lock on an existing file.

Parameters

file_name	The name of the file.
ref_fp	if successful, the file descriptor of the file on which we have an exclusive read/write lock.

Returns: If the file does not exist, return immediately indicating failure (false), otherwise block until an exclusive read/write lock can be obtained and then return true.

Exceptions

Error is thrown to indicate a number of untoward events.

Definition at line 201 of file BESFileLockingCache.cc.

References BESDEBUG.

Referenced by BESFileLockingCache::purge_file().

void BESFileLockingCache::initialize	(	const string &	cache_dir,
		const string &	prefix,
		unsigned long long	size
	)

protectedinherited

Definition at line 86 of file BESFileLockingCache.cc.

void BESFileLockingCache::lock_cache_read ( )

virtualinherited

Get a shared lock on the 'cache info' file.

Definition at line 540 of file BESFileLockingCache.cc.

References BESDEBUG.

Referenced by BESFileLockingCache::get_cache_size(), and BESFileLockingCache::get_read_lock().

void BESFileLockingCache::lock_cache_write ( )

virtualinherited

Get an exclusive lock on the 'cache info' file.

The 'cache info' file is used to control certain cache actions, ensuring that they are atomic. These include making sure that the create_and_lock() and read_and_lock() operations are atomic as well as the purge and related operations.

Note: This is intended to be used internally only but might be useful in some settings.

Definition at line 528 of file BESFileLockingCache.cc.

References BESDEBUG.

Referenced by BESFileLockingCache::create_and_lock(), BESFileLockingCache::purge_file(), BESFileLockingCache::update_and_purge(), and BESFileLockingCache::update_cache_info().

void BESFileLockingCache::purge_file ( const string & file )

virtualinherited

Purge a single file from the cache.

Purge a single file from the cache. The file might be old, etc., and need to be removed. Don't use this to shrink the cache when it gets too big, use update_and_purge() instead since that file optimizes accesses to the cache control file for several changes in a row.

Todo:: This is a new feature; add to BESCache3

Parameters

file	The name of the file to purge.

Definition at line 842 of file BESFileLockingCache.cc.

References BESDEBUG, BESFileLockingCache::get_cache_size(), BESFileLockingCache::getExclusiveLock(), BESFileLockingCache::lock_cache_write(), and BESFileLockingCache::unlock_cache().

Referenced by BESDapResponseCache::cache_dataset(), and BESH4Cache::get_data_from_cache().

Here is the call graph for this function:

string BESStoredDapResultCache::store_dap2_result	(	libdap::DDS &	dds,
		const std::string &	constraint,
		BESDapResponseBuilder *	rb,
		libdap::ConstraintEvaluator *	eval
	)

virtual

Returns: The local ID (relative to the BES data root directory) of the stored dataset.

FIXME I think we need to delete this here, before we close and unlock the cache.

Definition at line 692 of file BESStoredDapResultCache.cc.

References BESDEBUG, CRLF, BESDapResponseBuilder::get_dataset_name(), and BESDapResponseBuilder::serialize_dap2_data_ddx().

Referenced by BESDapResponseBuilder::store_dap2_result().

Here is the call graph for this function:

string BESStoredDapResultCache::store_dap4_result	(	libdap::DMR &	dmr,
		const string &	constraint,
		BESDapResponseBuilder *	rb
	)

virtual

Returns: The local ID (relative to the BES data root directory) of the stored dataset.

FIXME Is it cool to unlock and close the cache file at this point? I think so...

Definition at line 885 of file BESStoredDapResultCache.cc.

References BESDEBUG, and BESDapResponseBuilder::serialize_dap4_data().

Referenced by BESDapResponseBuilder::store_dap4_result().

Here is the call graph for this function:

void BESFileLockingCache::unlock_and_close ( const string & file_name )

virtualinherited

Unlock the named file.

This does not do any name mangling; it just closes and unlocks whatever is named (or throws BESBESInternalErroror if the file cannot be closed). If the file was opened more than once, all descriptors are closed. If you need to close a specific descriptor, use the other version of unlock_and_close().

Note: This method assumes that the file was opened/locked using one of read_and_lock() or create_and_lock(). Those methods record the name/file- descriptor pairs so that the files can be properly closed and locks released.

Parameters

file_name The name of the file to unlock.

Exceptions

BESBESInternalErroror

Definition at line 578 of file BESFileLockingCache.cc.

References BESDEBUG.

Referenced by BESH4Cache::get_data_from_cache(), BESFileContainer::release(), and BESH4Cache::write_cached_data().

void BESFileLockingCache::unlock_cache ( )

virtualinherited

Unlock the cache info file.

Note: This is intended to be used internally only bt might be useful in some settings.

Definition at line 554 of file BESFileLockingCache.cc.

References BESDEBUG.

Referenced by BESDapResponseCache::cache_dataset(), BESFileLockingCache::create_and_lock(), BESFileLockingCache::get_cache_size(), BESFileLockingCache::get_read_lock(), BESFileLockingCache::purge_file(), gateway::RemoteHttpResource::retrieveResource(), BESUncompressManager3::uncompress(), BESFileLockingCache::update_and_purge(), and BESFileLockingCache::update_cache_info().

void BESFileLockingCache::update_and_purge ( const string & new_file )

virtualinherited

Purge files from the cache.

Purge files, oldest to newest, if the current size of the cache exceeds the size of the cache specified in the constructor. This method uses an exclusive lock on the cache for the duration of the purge process.

Parameters

new_file The name of a file this process just added to the cache. Using fcntl(2) locking there is no way this process can detect its own lock, so the shared read lock on the new file won't keep this process from deleting it (but will keep other processes from deleting it).

Definition at line 758 of file BESFileLockingCache.cc.

References BESDEBUG, BESISDEBUG, BESFileLockingCache::cache_too_big(), BESFileLockingCache::lock_cache_write(), and BESFileLockingCache::unlock_cache().

Referenced by BESDapResponseCache::cache_dataset(), gateway::RemoteHttpResource::retrieveResource(), BESUncompressManager3::uncompress(), and BESH4Cache::write_cached_data().

Here is the call graph for this function:

unsigned long long BESFileLockingCache::update_cache_info ( const string & target )

virtualinherited

Update the cache info file to include 'target'.

Add the size of the named file to the total cache size recorded in the cache info file. The cache info file is exclusively locked by this method for its duration. This updates the cache info file and returns the new size.

Parameters

target The name of the file

Returns: The new size of the cache

Definition at line 621 of file BESFileLockingCache.cc.

References BESDEBUG, BESFileLockingCache::lock_cache_write(), and BESFileLockingCache::unlock_cache().

Referenced by BESDapResponseCache::cache_dataset(), gateway::RemoteHttpResource::retrieveResource(), BESUncompressManager3::uncompress(), and BESH4Cache::write_cached_data().

Here is the call graph for this function:

Friends And Related Function Documentation

friend class ResponseBuilderTest

friend

Definition at line 74 of file BESStoredDapResultCache.h.

friend class StoredDap2ResultTest

friend

Definition at line 72 of file BESStoredDapResultCache.h.

friend class StoredDap4ResultTest

friend

Definition at line 73 of file BESStoredDapResultCache.h.

Member Data Documentation

const string BESStoredDapResultCache::PREFIX_KEY = "DAP.StoredResultsCache.prefix"

static

Definition at line 89 of file BESStoredDapResultCache.h.

const string BESStoredDapResultCache::SIZE_KEY = "DAP.StoredResultsCache.size"

static

Definition at line 90 of file BESStoredDapResultCache.h.

const string BESStoredDapResultCache::SUBDIR_KEY = "DAP.StoredResultsCache.subdir"

static

Definition at line 88 of file BESStoredDapResultCache.h.

The documentation for this class was generated from the following files:

Public Member Functions

Static Public Member Functions

Static Public Attributes

Protected Member Functions

Friends

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Friends And Related Function Documentation

Member Data Documentation