CMU 15-445/645 PROJECT #1 - BUFFER POOL上(实现线程安全的LRU)
LRU的实现,实际上就是在算法题的基础上,加上线程完全。
用C++来写的话,可以使用哈希表和list,不用自己去实现双向链表
线程安全直接用C++11锁来做
缓冲buffer未完待续。
During the semester, you will be building a new disk-oriented storage manager for the BusTub DBMS. Such a storage manager assumes that the primary storage location of the database is on disk.
The first programming project is to implement a buffer pool in your storage manager. The buffer pool is responsible for moving physical pages back and forth from main memory to disk. It allows a DBMS to support databases that are larger than the amount of memory that is available to the system. The buffer pool's operations are transparent to other parts in the system. For example, the system asks the buffer pool for a page using its unique identifier (page_id_t
) and it does not know whether that page is already in memory or whether the system has to go retrieve it from disk.
Your implementation will need to be thread-safe. Multiple threads will be accessing the internal data structures at the same and thus you need to make sure that their critical sections are protected with latches (these are called "locks" in operating systems).
You will need to implement the following two components in your storage manager:
- LRU Replacement Policy
- Buffer Pool Manager
This is a single-person project that will be completed individually (i.e. no groups).
- Release Date: Sep 14, 2020
- Due Date: Sep 27, 2020 @ 11:59pm
For each of the following components, we are providing you with stub classes that contain the API that you need to implement. You should not modify the signatures for the pre-defined functions in these classes. If you modify the signatures, the test code that we use for grading will break and you will get no credit for the project. You also should not add additional classes in the source code for these components. These components should be entirely self-contained.
PROJECT SPECIFICATION
For each of the following components, we are providing you with stub classes that contain the API that you need to implement. You should not modify the signatures for the pre-defined functions in these classes. If you modify the signatures, the test code that we use for grading will break and you will get no credit for the project. You also should not add additional classes in the source code for these components. These components should be entirely self-contained.
If a class already contains data members, you should not remove them. For example, the BufferPoolManager
contains DiskManager
and Replacer
objects. These are required to implement the functionality that is needed by the rest of the system. On the other hand, you may need to add data members to these classes in order to correctly implement the required functionality. You can also add additional helper functions to these classes. The choice is yours.
You are allowed to use any built-in C++17 containers in your project unless specified otherwise. It is up to you to decide which ones you want to use. Note that these containers are not thread-safe and that you will need to include latches in your implementation to protect them. You may not bring in additional third-party dependencies (e.g. boost).
TASK #1 - LRU REPLACEMENT POLICY
This component is responsible for tracking page usage in the buffer pool. You will implement a new sub-class called LRUReplacer
in src/include/buffer/lru_replacer.h and its corresponding implementation file in src/buffer/lru_replacer.cpp. LRUReplacer
extends the abstract Replacer
class (src/include/buffer/replacer.h), which contains the function specifications.
The size of the LRUReplacer
is the same as buffer pool since it contains placeholders for all of the frames in the BufferPoolManager
. However, not all the frames are considered as in the LRUReplacer
. The LRUReplacer
is initialized to have no frame in it. Then, only the newly unpinned ones will be considered in the LRUReplacer
.
You will need to implement the LRU policy discussed in the class. You will need to implement the following methods:
Victim(T*)
: Remove the object that was accessed the least recently compared to all the elements being tracked by theReplacer
, store its contents in the output parameter and returnTrue
. If theReplacer
is empty returnFalse
.Pin(T)
: This method should be called after a page is pinned to a frame in theBufferPoolManager
. It should remove the frame containing the pinned page from theLRUReplacer
.Unpin(T)
: This method should be called when thepin_count
of a page becomes 0. This method should add the frame containing the unpinned page to theLRUReplacer
.Size()
: This method returns the number of frames that are currently in theLRUReplacer
.
The implementation details are up to you. You are allowed to use built-in STL containers. You can assume that you will not run out of memory, but you must make sure that the operations are thread-safe.
LRU的实现,建议使用C++ STL unordered_map和list,在外面加一把大锁即可。
注意这些方法的定义和Leetcode上题目描述不同,结合测试用例和英文原文理解。
namespace bustub {/*** LRUReplacer implements the lru replacement policy, which approximates the Least Recently Used policy.*/
class LRUReplacer : public Replacer {public:/*** Create a new LRUReplacer.* @param num_pages the maximum number of pages the LRUReplacer will be required to store*/explicit LRUReplacer(size_t num_pages);/*** Destroys the LRUReplacer.*/~LRUReplacer() override;bool Victim(frame_id_t *frame_id) override;void Pin(frame_id_t frame_id) override;void Unpin(frame_id_t frame_id) override;size_t Size() override;private:using ListIterator = typename std::list<frame_id_t>::const_iterator;using CacheMap = std::unordered_map<frame_id_t,ListIterator>;size_t num_pages;std::list<frame_id_t> m_lruList;CacheMap m_lruMap;std::mutex m_lock;
};} // namespace bustub
namespace bustub {LRUReplacer::LRUReplacer(size_t num_pages):num_pages(num_pages) {m_lruMap.reserve(this->num_pages);
}LRUReplacer::~LRUReplacer() = default;bool LRUReplacer::Victim(frame_id_t *frame_id) {std::lock_guard<std::mutex> guard(m_lock);if(Size()==0) { return false;}*frame_id = m_lruList.back();m_lruMap.erase(m_lruList.back());m_lruList.pop_back();return true;
}void LRUReplacer::Pin(frame_id_t frame_id) {std::lock_guard<std::mutex> guard(m_lock);if(m_lruMap.find(frame_id)==m_lruMap.end()) { return;}m_lruList.erase(m_lruMap[frame_id]);m_lruMap.erase(frame_id);
}void LRUReplacer::Unpin(frame_id_t frame_id) {std::lock_guard<std::mutex> guard(m_lock);// key already exists, just update the queueif(auto mapIter = m_lruMap.find(frame_id);mapIter!=m_lruMap.end()){//m_lruList.splice(m_lruList.begin(),m_lruList,mapIter->second);}else{m_lruList.push_front(frame_id);m_lruMap[frame_id] = m_lruList.begin();}
}size_t LRUReplacer::Size() { return m_lruMap.size(); }} // namespace bustub
然后是测试代码
namespace bustub {TEST(LRUReplacerTest, SampleTest) {LRUReplacer lru_replacer(7);// Scenario: unpin six elements, i.e. add them to the replacer.lru_replacer.Unpin(1);lru_replacer.Unpin(2);lru_replacer.Unpin(3);lru_replacer.Unpin(4);lru_replacer.Unpin(5);lru_replacer.Unpin(6);lru_replacer.Unpin(1);EXPECT_EQ(6, lru_replacer.Size());// Scenario: get three victims from the lru.int value;lru_replacer.Victim(&value);EXPECT_EQ(1, value);lru_replacer.Victim(&value);EXPECT_EQ(2, value);lru_replacer.Victim(&value);EXPECT_EQ(3, value);// Scenario: pin elements in the replacer.// Note that 3 has already been victimized, so pinning 3 should have no effect.lru_replacer.Pin(3);lru_replacer.Pin(4);EXPECT_EQ(2, lru_replacer.Size());// Scenario: unpin 4. We expect that the reference bit of 4 will be set to 1.lru_replacer.Unpin(4);// Scenario: continue looking for victims. We expect these victims.lru_replacer.Victim(&value);EXPECT_EQ(5, value);lru_replacer.Victim(&value);EXPECT_EQ(6, value);lru_replacer.Victim(&value);EXPECT_EQ(4, value);
}} // namespace bustub
TASK #2 - BUFFER POOL MANAGER
Next, you need to implement the buffer pool manager in your system (BufferPoolManager
). The BufferPoolManager
is responsible for fetching database pages from the DiskManager
and storing them in memory. The BufferPoolManager
can also write dirty pages out to disk when it is either explicitly instructed to do so or when it needs to evict a page to make space for a new page.
To make sure that your implementation works correctly with the rest of the system, we will provide you with some of the functions already filled in. You will also not need to implement the code that actually reads and writes data to disk (this is called the DiskManager
in our implementation). We will provide that functionality for you.
All in-memory pages in the system are represented by Page
objects. The BufferPoolManager
does not need to understand the contents of these pages. But it is important for you as the system developer to understand that Page
objects are just containers for memory in the buffer pool and thus are not specific to a unique page. That is, each Page
object contains a block of memory that the DiskManager
will use as a location to copy the contents of a physical page that it reads from disk. The BufferPoolManager
will reuse the same Page
object to store data as it moves back and forth to disk. This means that the same Page
object may contain a different physical page throughout the life of the system. The Page
object's identifer (page_id
) keeps track of what physical page it contains; if a Page
object does not contain a physical page, then its page_id
must be set to INVALID_PAGE_ID
.
Each Page
object also maintains a counter for the number of threads that have "pinned" that page. Your BufferPoolManager
is not allowed to free a Page
that is pinned. Each Page
object also keeps track of whether it is dirty or not. It is your job to record whether a page was modified before it is unpinned. Your BufferPoolManager
must write the contents of a dirty Page
back to disk before that object can be reused.
Your BufferPoolManager
implementation will use the LRUReplacer
class that you created in the previous steps of this assignment. It will use the LRUReplacer
to keep track of when Page
objects are accessed so that it can decide which one to evict when it must free a frame to make room for copying a new physical page from disk.
You will need to implement the following functions defined in the header file (src/include/buffer/buffer_pool_manager.h) in the source file (src/buffer/buffer_pool_manager.cpp):
FetchPageImpl(page_id)
NewPageImpl(page_id)
UnpinPageImpl(page_id, is_dirty)
FlushPageImpl(page_id)
DeletePageImpl(page_id)
FlushAllPagesImpl()
For FetchPageImpl,you should return NULL if no page is available in the free list and all other pages are currently pinned. FlushPageImpl should flush a page regardless of its pin status.
Refer to the function documentation for details on how to implement these functions. Don't touch the non-impl versions, we need those to grade your code.
未完待续
CMU 15-445/645 PROJECT #1 - BUFFER POOL上(实现线程安全的LRU)相关推荐
- CMU15-445 PROJECT #1 - BUFFER POOL(Fall2020实验代码,已满分)
实验说明:https://15445.courses.cs.cmu.edu/fall2020/project1 我的完整实验代码见github:https://github.com/nefu-ljw/ ...
- CMU 15-445实验记录(二):Project 1 Buffer Pool Manager
CMU 15-445实验记录(二):Project 1 在project 1中,已经为我们提供了磁盘管理器以及page layouts ,我们要构建自己的buffer pool管理器以及替换 策略,根 ...
- CMU15445 buffer pool 2021
CMU15445 buffer pool 2021 心得体会 Gradescope测试 源代码(仅供参考) LRU Replacement Policy Buffer Pool Manager Ins ...
- MySQL · 性能优化· InnoDB buffer pool flush策略漫谈
MySQL · 性能优化· InnoDB buffer pool flush策略漫谈 背景 我们知道InnoDB使用buffer pool来缓存从磁盘读取到内存的数据页.buffer pool通常由数 ...
- 重学MySQL(InnoDB Buffer Pool是什么?)
文章目录 InnoDB Buffer Pool是什么? 我们的数据是如何放在InnoDB Buffer Pool中的? InnoDB怎么知道数据页是否在Buffer Pool中? InnoDB Buf ...
- buffer pool详解(free链表+flush链表+lru链表)
buffer pool是什么? 是一块内存区域,当数据库操作数据的时候,把硬盘上的数据加载到buffer pool,不直接和硬盘打交道,操作的是buffer pool里面的数据 数据库的增删改查都是在 ...
- 从一条sql的执行流程来详细了解Buffer Pool
一条更新语句的执行流程 为什么Mysql不能直接更新磁盘上的数据而且设置这么一套复杂的机制来执行SQL? 因为来一个请求就直接对磁盘文件进行随机读写,然后更新磁盘文件里的数据性能可能相当差. 因为磁盘 ...
- MySQL怎么运行的系列(二)Innodb缓冲池 buffer pool 和 改良版LRU算法
本系列文章目录 展开/收起 MySQL怎么运行的系列(一)mysql体系结构和存储引擎 MySQL怎么运行的系列(二)Innodb缓冲池 buffer pool 和 改良版LRU算法 MySQL怎么运 ...
- MySQL8.0 存储引擎(InnoDB )buffer pool的实现原理
数据库为了高效读取和存储物理数据,通常都会采用缓存的方式来弥补磁盘IO与CPU运算速度差.InnoDB 作为一个具有高可靠性和高性能的通用存储引擎也不例外,Buffer Pool就是其用来在内存中缓存 ...
最新文章
- 对Julia社区不熟悉?创始人来告诉你
- docker helowin 迁移_docker-compose 安装 oracle_11g_r2 并实现数据持久化
- 一篇文章让你轻松搞定SpringBoot和SpringCloud之间的版本选择!!!
- diffpatch升级_Linux diff创建补丁以及patch打补丁
- 解释器android,Android的设计模式-解释器模式
- c语言实现可变单链表,c语言实现单链表
- python is 与 == 的区别
- UVa 12657 双向链表
- JVM面试重点总结(一)——java内存区域与内存溢出异常
- 空巢青年,“空巢”是选择还是无奈? | 数据告诉你
- 大一高数下册笔记整理_高等数学下册知识点总结.doc
- Java的静态数组和动态数组
- html有多少种居中方式,html常用的几种居中方法
- plc secs通讯协议_一种SECS/GEM通讯方法与流程
- 【转载】2021互联网行业术语@20210311
- java月份下拉菜单_实现一个日期下拉菜单
- 循序渐进理解RTFM算法
- 区块链技术方向的就业前景
- 转:网络安全法重点解读
- 助力零售业降本增效,零售业相关场景RPA应用