难度: Medium
原题连接
内容描述
Given a list of directory info including directory path, and all the files with contents in this directory, you need to find out all the groups of duplicate files in the file system in terms of their paths.
A group of duplicate files consists of at least two files that have exactly the same content.
A single directory info string in the input list has the following format:
"root/d1/d2/.../dm f1.txt(f1_content) f2.txt(f2_content) ... fn.txt(fn_content)"
It means there are n files (f1.txt, f2.txt ... fn.txt with content f1_content, f2_content ... fn_content, respectively) in directory root/d1/d2/.../dm. Note that n >= 1 and m >= 0. If m = 0, it means the directory is just the root directory.
The output is a list of group of duplicate file paths. For each group, it contains all the file paths of the files that have the same content. A file path is a string that has the following format:
"directory_path/file_name.txt"
Example 1:
Input:
["root/a 1.txt(abcd) 2.txt(efgh)", "root/c 3.txt(abcd)", "root/c/d 4.txt(efgh)", "root 4.txt(efgh)"]
Output:
[["root/a/2.txt","root/c/d/4.txt","root/4.txt"],["root/a/1.txt","root/c/3.txt"]]
Note:
No order is required for the final output.
You may assume the directory name, file name and file content only has letters and digits, and the length of file content is in the range of [1,50].
The number of files given is in the range of [1,20000].
You may assume no files or directories share the same name in the same directory.
You may assume each given directory info represents a unique directory. Directory path and file info are separated by a single blank space.
Follow-up beyond contest:
Imagine you are given a real file system, how will you search files? DFS or BFS?
If the file content is very large (GB level), how will you modify your solution?
If you can only read the file by 1kb each time, how will you modify your solution?
What is the time complexity of your modified solution? What is the most time-consuming part and memory consuming part of it? How to optimize?
How to make sure the duplicated files you find are not false positive?
思路 1 - 时间复杂度: O(sum(len(path) for path in paths))- 空间复杂度: O(sum(len(path) for path in paths))******
用content作为key来存放，最后相同content拥有超过一个file的就作为结果返回
beats 25.00%
class Solution:
def findDuplicate(self, paths):
"""
:type paths: List[str]
:rtype: List[List[str]]
"""
lookup = collections.defaultdict(list)
for path in paths:
lst = path.split(' ')
prefix = lst[0] + '/'
for file in lst[1:]:
idx = file.find('(')
content = file[idx+1:-1]
lookup[content].append(prefix+file[:idx])
res = []
for k, v in lookup.items():
if len(v) > 1:
res.append(v)
return res写得简单点，beats 44.91%
class Solution:
def findDuplicate(self, paths):
"""
:type paths: List[str]
:rtype: List[List[str]]
"""
lookup = collections.defaultdict(list)
for path in paths:
root, *f = path.split(" ")
for file in f:
file_name, content = file.split('(')
lookup[content].append(root+'/'+file_name)
return [lookup[k] for k in lookup if len(lookup[k]) > 1]参考Follow up questions discussion
- Imagine you are given a real file system, how will you search files? DFS or BFS?
DFS，因为DFS更省内存，并且更容易实现
- If the file content is very large (GB level), how will you modify your solution?
content非常大，那么我就会先用content的size来作为key存，然后在content size相同的所有file里面，找出对应content hash相同的file， 它们肯定就是duplicate了
One approach is to also maintain a metadata field for each file's size on disk. Then you can take a 2 pass approach:
- DFS to map each size to a set of paths that have that size
- For each size, if there are more than 2 files there, compute hash of every file, if any files with the same size have the same hash, then they are identical files.
To optimize Step-2. In GFS, it stores large file in multiple "chunks" (one chunk is 64KB). we have meta data, including the file size, file name and index of different chunks along with each chunk's checkSum(the xor for the content). For step-2, we just compare each file's checkSum. Disadvantage: there might be flase positive duplicates, because two different files might share the same checkSum.
- If you can only read the file by 1kb each time, how will you modify your solution?
- makeHashQuick Function is quick but memory hungry, might likely to run with java -Xmx2G or the likely to increase heap space if RAM avaliable.
- we might need to play with the size defined by "buffSize" to make memory efficient.
- What is the time complexity of your modified solution? What is the most time-consuming part and memory consuming part of it? How to optimize?
- hashing part is the most time-consuming and memory consuming.
- question 2中的方法基本避免了false positive问题，因为content size相等，hash相同，并且每个chunk id都相同，这样的file不相同的概率太小了