Date:   Tue, 19 Jan 2021 14:39:28 +0530
From:   Abhijit Menon-Sen <ams@...oid.org>
To:     linux-ext4@...r.kernel.org
Subject: advice on recovery from fs corruption

Hi.

Summary: I have an ext4 filesystem on a LUKS-encrypted device, and I
carelessly overwrote the first ~2.5GB of the underlying block device
with dd(1). While chastising myself for being so unforgivably careless,
I humbly request advice on trying to recover anything from the corrupted
filesystem, which is still mounted.

Here's the block device and filesystem in question:

    NAME           MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
    sdb              8:16   0  7.2T  0 disk
    └─sdb1           8:17   0  7.2T  0 part
      └─sdb1_crypt 254:2    0  7.2T  0 crypt /media/nas/l1

    /dev/mapper/sdb1_crypt is active and is in use.
      type:    n/a
      cipher:  aes-xts-plain64
      keysize: 256 bits
      device:  /dev/sdb1
      offset:  4096 sectors
      size:    15493750784 sectors
      mode:    read/write

    Filesystem              1K-blocks       Used  Available Use% Mounted on
    /dev/mapper/sdb1_crypt 7696239772 3081218516 4615021256  41% /media/nas/l1

I ran `dd if=ubuntu.iso of=/dev/sdb bs=8M status=progress oflag=direct`,
which wrote 2715254784 bytes to /dev/sdb.

The device is still mounted, and as soon as I realised that I ran this
command on the wrong machine (it had already completed), I started to
rsync a couple of directories off onto another volume.

One rsync is still copying files, but now "ls" under /media/nas/l1 shows
nothing, not even lost+found. The cp -a I started finished successfully
(with I/O errors on a few files), but that shell also sees no other
files now. dmesg shows many errors like this:

    [1185480.158594] EXT4-fs error (device dm-2): ext4_get_branch:171: inode #81696398: block 2403213648: comm cp: invalid block
    [1185480.159125] EXT4-fs error (device dm-2): ext4_map_blocks:605: inode #81696398: block 2403213648: comm cp: lblock 13 mapped to illegal pblock 2403213648 (length 1)
    …
    [1187747.065781] EXT4-fs error (device dm-2): htree_dirblock_to_tree:1023: inode #2: block 1239: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0, inode=1002371330, rec_len=24822, name_len=20, size=4096

Along with the superblock, I guess the block group descriptors were
overwritten, and enough of the inode tables that it can't find the root
directory or lost+found any more. Definitely not the recommended way to
install Ubuntu.

Here's what `dumpe2fs -h /dev/mapper/sdb1_crypt` shows. (Is it getting
this information from one of the backup superblocks?)

    dumpe2fs 1.44.5 (15-Dec-2018)
    Filesystem volume name:   <none>
    Last mounted on:          /media/nas/l1
    Filesystem UUID:          602da5a3-9b1d-4a44-a55f-60e333b107cd
    Filesystem magic number:  0xEF53
    Filesystem revision #:    1 (dynamic)
    Filesystem features:      ext_attr resize_inode dir_index filetype sparse_super large_file
    Filesystem flags:         signed_directory_hash
    Default mount options:    user_xattr acl
    Filesystem state:         not clean with errors
    Errors behavior:          Continue
    Filesystem OS type:       Linux
    Inode count:              200480768
    Block count:              1936718848
    Reserved block count:     0
    Free blocks:              1153755314
    Free inodes:              198450506
    First block:              0
    Block size:               4096
    Fragment size:            4096
    Reserved GDT blocks:      562
    Blocks per group:         32768
    Fragments per group:      32768
    Inodes per group:         3392
    Inode blocks per group:   212
    Filesystem created:       Sun Jan 20 22:32:01 2019
    Last mount time:          Tue Jan  5 20:29:29 2021
    Last write time:          Tue Jan 19 14:05:06 2021
    Mount count:              1
    Maximum mount count:      -1
    Last checked:             Tue Jan  5 19:26:10 2021
    Check interval:           0 (<none>)
    Lifetime writes:          2744 GB
    Reserved blocks uid:      0 (user root)
    Reserved blocks gid:      0 (group root)
    First inode:              11
    Inode size:	          256
    Required extra isize:     32
    Desired extra isize:      32
    Default directory hash:   half_md4
    Directory Hash Seed:      f0788ffe-7f52-404b-8395-e6e219da154e
    FS Error count:           17307
    First error time:         Wed Jan  6 07:56:19 2021
    First error function:     ext4_write_inode
    First error line #:       5463
    First error inode #:      163541339
    First error block #:      1579843763
    Last error time:          Tue Jan 19 14:05:06 2021
    Last error function:      htree_dirblock_to_tree
    Last error line #:        1023
    Last error inode #:       2
    Last error block #:       1239
    ext2fs_read_bb_inode: Cannot iterate data blocks of an inode containing inline data

A large portion of the contents of this filesystem are, unfortunately,
irreplaceable. The data were protected only from disk failure, not from
this sort of operator error, and I certainly regret making that decision
years ago.

I'm running `dd if=/dev/mapper/sdb1_crypt of=sdb1_crypt.img bs=8M
status=progress` now to capture an image of the unencrypted blocks of
the filesystem. I can't unmount the filesystem or anything, because I'll
never be able to mount it again. I don't even have a backup of the LUKS
header, nor an e2image for this filesystem.

It'll take a couple of days to copy all of the blocks from sdb1_crypt
(assuming nothing blows up in the interim). I'm willing to go to great
lengths to try to recover some of the data (including writing code to
trawl around the filesystem image, if needed).

With sincere apologies for asking other people to spend time on my
self-inflicted problem… what is the best approach to recovering as
many of my files as possible?

Thank you.

-- ams

Powered by blists - more mailing lists