aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorAlex Auvolat <alex@adnab.me>2018-11-06 11:57:09 +0100
committerAlex Auvolat <alex@adnab.me>2018-11-06 11:57:09 +0100
commitc4f6cbab20b0b1d08755073d93365e5bd00dc755 (patch)
tree384dd05c59a810d503790a2af1b5c49e1aed5ccf
parent0e5b82f3348508eb7f4291a08722522a49edd752 (diff)
downloadshard-c4f6cbab20b0b1d08755073d93365e5bd00dc755.tar.gz
shard-c4f6cbab20b0b1d08755073d93365e5bd00dc755.zip
Shard deletion ; update README and TODO
-rw-r--r--README.md102
-rw-r--r--TODO31
-rw-r--r--shard/lib/app/chat.ex5
-rw-r--r--shard/lib/app/directory.ex4
-rw-r--r--shard/lib/app/file.ex6
-rw-r--r--shard/lib/app/identity.ex4
-rw-r--r--shard/lib/app/pagestore.ex16
-rw-r--r--shard/lib/cli/cli.ex10
-rw-r--r--shard/lib/manager.ex31
-rw-r--r--shardweb/lib/controllers/directory_controller.ex6
-rw-r--r--shardweb/lib/controllers/page_controller.ex8
-rw-r--r--shardweb/lib/router.ex1
-rw-r--r--shardweb/lib/templates/page/shard_list.html.eex15
13 files changed, 175 insertions, 64 deletions
diff --git a/README.md b/README.md
index 107d3bd..c247e4e 100644
--- a/README.md
+++ b/README.md
@@ -1,10 +1,10 @@
What is Shard?
===============
-*WARNING:* This is a primer on what I want Shard to become. As of today I'm
+**WARNING: This is a primer on what I want Shard to become. As of today I'm
only starting work on this project and there is almost nothing to see. I hope
this will evolve and approach at least a significant portion of the goals
-identified below.
+identified below.**
Shard is a decentralized communication system adapted to public and private
discussion and collaboration.
@@ -48,9 +48,9 @@ significant manners:
Shard vs. Secure Scuttlebutt
----------------------------
-Just like SSB, Shard will enable users to work off grid and exchange data
-when they happen to communicate. We wish to improve upon the following
-aspects of SSB:
+Just like SSB, Shard will enable users to work off grid and exchange data when
+a network connection happens to be available. We wish to improve upon the
+following aspects of SSB:
- More granularity on what peers store or not, by splitting a user's
contribution in different "shards" that correspond to different topics.
@@ -63,7 +63,7 @@ aspects of SSB:
new people and finding contents.
- We wish to be very general from the start, and not limited to social-media-like
conversations.
-- We want excellent support for one-to-one and groupwise communications.
+- We want excellent support for private one-to-one and groupwise communications.
Shard vs. Matrix
----------------
@@ -80,6 +80,27 @@ Shard vs. Mastodon / GNU Social
Same remarks than for Matrix.
+Current status
+==============
+
+What is available
+-----------------
+
+* Chat rooms (public and private) with full history and efficient data structure for
+ retrieving missing messages after disconnection
+* File upload (public only)
+* Directories as collections of links to shards
+
+What is missing
+---------------
+
+See `TODO` file for more details.
+
+* Finding peers via DHT (very easy to add)
+* Invite/notification system
+* Good access control
+* More applications
+
How to use it?
==============
@@ -103,61 +124,77 @@ Then, clone the Shard repository using git:
git clone git://adnab.me/shard.git
```
-### CLI version only
-Enter the shard directory and create a data folder.
+### Web UI version
+
+Enter the shard directory and create a data2 folder.
```
cd shard
-mkdir data # for CLI use only
+mkdir data2 # for web UI
```
-Enter the shard subdirectory and install Elixir dependencies.
+Enter the shardweb subdirectory and install Elixir dependencies.
```
-cd shard
+cd shardweb
mix deps.get
mix compile
```
+Build the web assets:
+
+```
+cd assets
+npm install
+cd ..
+```
+
**The default configuration assumes you have cloned the shard repo in your
-`$HOME`. If this is not the case, ajust `config/config.exs` accordingly.**
+`$HOME`. If this is not the case, ajust the data path parameter in
+`shardweb/config/config.exs` accordingly.**
-### Web UI version
+### CLI version only
-Enter the shard directory and create a data2 folder.
+Enter the shard directory and create a data folder.
```
cd shard
-mkdir data2 # for web UI
+mkdir data # for CLI use only
```
-Enter the shardweb subdirectory and install Elixir dependencies.
+Enter the shard subdirectory and install Elixir dependencies.
```
-cd shardweb
+cd shard
mix deps.get
mix compile
```
-Build the web assets:
+**The default configuration assumes you have cloned the shard repo in your
+`$HOME`. If this is not the case, ajust the data path parameter in
+`shard/config/config.exs` accordingly.**
+
+
+Web Ui usage
+------------
+
+Once you have completed the installation steps above, Shard web can be started
+by running the following in the `shardweb` directory of the repo:
```
-cd assets
-npm install
-cd ..
+iex -S mix phx.server
```
-**The default configuration assumes you have cloned the shard repo in your
-`$HOME`. If this is not the case, ajust `config/config.exs` accordingly.**
+The UI will become accessible at address `http://127.0.0.1:4000`.
CLI usage
---------
Once you have completed the installation steps above, Shard can be started by
-running the following in the `shard/shard` directory:
+running the following in the `shard` directory of the repo:
```
iex -S mix
@@ -177,18 +214,7 @@ This CLI supports a few basic commands:
- `/nick my_nickname`: set nickname
- `/list`: list channels we are currently connected to
- `/hist`: show recent messages on current channel
-- `/pm nickname`: enter private conversation with someone
+- `/pm nickname1 [nickname2] [...]`: enter private conversation with someone
+- `/send_file path`: make file available on the network and send link to current chat room.
+ **WARNING: all files are publicly available for now, even if they are sent in a private chat room.**
- `/quit`: return to iex prompt
-
-
-Web Ui usage
-------------
-
-Once you have completed the installation steps above, Shard web can be started
-by running the following in the `shard/shardweb` directory:
-
-```
-iex -S mix phx.server
-```
-
-The UI will become accessible at address `http://127.0.0.1:4000`.
diff --git a/TODO b/TODO
index dd13e1d..baf34f7 100644
--- a/TODO
+++ b/TODO
@@ -10,13 +10,16 @@ achieves our goals.
TASK LIST
=========
-- High priority: invite
-- Medium priority: dht, dep
+- High priority: invite, dht, groups
+- Medium priority: net
-Invitation system (invite, EASY)
+Invitation system (invite, MED)
--------------------------------
+Current issue: if a user PM's another user, that user does not know it!
+He has to pull the PM shard voluntarily to get the messages.
+
A user may generate invitation tokens: an invite token is a signature of the
invited user's pk by the inviter user's sk. The user that redeems the token
writes some data that is saved temporarily in the user's identity shard, which
@@ -39,7 +42,6 @@ Ex: keep 100 total open connections that are sampled by proximity on the set of
plus 2 or 5 full random for all shards.
-
DHT to find peers for a given shard (dht, EASY)
-----------------------------------
@@ -99,16 +101,6 @@ access control decisions (obviously these can only run when an
identity with admin privilege is running).
-Shard lifetime and dependency management (dep, MED)
-----------------------------------------
-
-Some Shards may pull other shards in, under certain conditions. For example
-a stored folder shard will just be a list of other shards that we all pull in.
-
-We want a way to have toplevel shards, shards that are dependencies of
-toplevel shards, and shards that we keep for a number of days as a cache but
-will expire automatically.
-
COMPLETED TASKS
===============
@@ -187,3 +179,14 @@ Best option: those that we are connected to + some random to
reach a quota (for example 10 or so)
Implementation: netgroups (lib/net/groups.ex)
+
+
+Shard lifetime and dependency management (dep, MED)
+----------------------------------------
+
+Some Shards may pull other shards in, under certain conditions. For example
+a stored folder shard will just be a list of other shards that we all pull in.
+
+We want a way to have toplevel shards, shards that are dependencies of
+toplevel shards, and shards that we keep for a number of days as a cache but
+will expire automatically.
diff --git a/shard/lib/app/chat.ex b/shard/lib/app/chat.ex
index 2795153..ff0c97d 100644
--- a/shard/lib/app/chat.ex
+++ b/shard/lib/app/chat.ex
@@ -131,6 +131,11 @@ defmodule SApp.Chat do
{:reply, state.manifest, state}
end
+ def handle_call(:delete_shard, _from, state) do
+ GenServer.call(state.store, :delete_store)
+ {:stop, :normal, :ok, state}
+ end
+
def handle_call({:read_history, top_bound, num}, _from, state) do
ret = MST.last(state.mst, top_bound, num)
{:reply, ret, state}
diff --git a/shard/lib/app/directory.ex b/shard/lib/app/directory.ex
index e9482f7..cbea8c3 100644
--- a/shard/lib/app/directory.ex
+++ b/shard/lib/app/directory.ex
@@ -64,6 +64,10 @@ defmodule SApp.Directory do
{:reply, state.manifest, state}
end
+ def handle_call(:delete_shard, _from, state) do
+ {:stop, :normal, :ok, state}
+ end
+
def handle_call(:get_items, _from, state) do
{:reply, SData.SignRev.get(state.items), state}
end
diff --git a/shard/lib/app/file.ex b/shard/lib/app/file.ex
index ce28beb..e2a9798 100644
--- a/shard/lib/app/file.ex
+++ b/shard/lib/app/file.ex
@@ -84,6 +84,12 @@ defmodule SApp.File do
{:reply, state.manifest, state}
end
+ def handle_call(:delete_shard, _from, state) do
+ GenServer.call(state.store, :delete_store)
+ File.rm(state.path)
+ {:stop, :normal, :ok, state}
+ end
+
def handle_call(:get_info, _from, state) do
reply = cond do
state.info != nil and GenServer.call(state.store, {:have_rec, state.info.merkle_root}) ->
diff --git a/shard/lib/app/identity.ex b/shard/lib/app/identity.ex
index 59a4b90..78abbe7 100644
--- a/shard/lib/app/identity.ex
+++ b/shard/lib/app/identity.ex
@@ -65,6 +65,10 @@ defmodule SApp.Identity do
{:reply, state.manifest, state}
end
+ def handle_call(:delete_shard, _from, state) do
+ {:stop, :normal, :ok, state}
+ end
+
def handle_call(:get_info, _from, state) do
{:reply, SData.SignRev.get(state.state), state}
end
diff --git a/shard/lib/app/pagestore.ex b/shard/lib/app/pagestore.ex
index 5165b84..3cda51d 100644
--- a/shard/lib/app/pagestore.ex
+++ b/shard/lib/app/pagestore.ex
@@ -25,7 +25,7 @@ defmodule SApp.PageStore do
@max_failures 4 # Maximum of peers that reply not_found before we abandon
defmodule State do
- defstruct [:shard_id, :path, :netgroup, :store, :reqs, :retries]
+ defstruct [:shard_id, :path, :netgroup, :store, :reqs, :retries, :store_path]
end
@@ -36,15 +36,21 @@ defmodule SApp.PageStore do
def init([shard_id, path, netgroup]) do
Shard.Manager.dispatch_to(shard_id, path, self())
- store_path = [Application.get_env(:shard, :data_path), "#{shard_id|>Base.encode16}.#{path}"] |> Path.join |> String.to_atom
- {:ok, store} = :dets.open_file store_path, [type: :set]
+ store_path = [Application.get_env(:shard, :data_path), "#{shard_id|>Base.encode16}.#{path}"] |> Path.join
+ {:ok, store} = :dets.open_file(String.to_atom(store_path), [type: :set])
Process.send_after(self(), :clean_cache, 1000)
- {:ok, %State{shard_id: shard_id, path: path, netgroup: netgroup, store: store, reqs: %{}, retries: %{}}}
+ {:ok, %State{shard_id: shard_id, path: path, netgroup: netgroup, store: store, reqs: %{}, retries: %{}, store_path: store_path}}
end
+ def handle_call(:delete_store, _from, state) do
+ :dets.close state.store
+ File.rm state.store_path
+ {:stop, :normal, :ok, state}
+ end
+
def handle_call({:get, key, prefer_ask}, from, state) do
case :dets.lookup state.store, key do
[{_, _, bin}] ->
@@ -84,9 +90,11 @@ defmodule SApp.PageStore do
case :dets.lookup state.store, hash do
[] ->
:dets.insert state.store, {hash, {:cached, System.os_time(:seconds) + @cache_ttl}, bin}
+ :dets.sync state.store
nil
[{_, why}] ->
:dets.insert state.store, {hash, why, bin}
+ :dets.sync state.store
why
[{_, _, _}] ->
nil
diff --git a/shard/lib/cli/cli.ex b/shard/lib/cli/cli.ex
index 380282d..54b882f 100644
--- a/shard/lib/cli/cli.ex
+++ b/shard/lib/cli/cli.ex
@@ -94,8 +94,16 @@ defmodule SCLI do
end
defp handle_command(state, ["list"]) do
- IO.puts "List of known channels:"
+ IO.puts "Private conversations:"
+ for {_chid, %SApp.Chat.PrivChat.Manifest{pk_list: pk_list}, _} <- Shard.Manager.list_shards do
+ pk_list
+ |> Enum.filter(&(&1 != state.pk))
+ |> Enum.map(fn pk -> "#{SApp.Identity.get_nick pk} #{Shard.Keys.pk_display pk}" end)
+ |> Enum.join(", ")
+ |> IO.puts
+ end
+ IO.puts "Public channels we are connected to:"
for {_chid, %SApp.Chat.Manifest{channel: chan}, _} <- Shard.Manager.list_shards do
IO.puts "##{chan}"
end
diff --git a/shard/lib/manager.ex b/shard/lib/manager.ex
index c178e2f..c3897a3 100644
--- a/shard/lib/manager.ex
+++ b/shard/lib/manager.ex
@@ -82,6 +82,27 @@ defmodule Shard.Manager do
{:reply, pid, state}
end
+ def handle_call({:delete, shard_id}, _from, state) do
+ case :dets.lookup(@shard_db, shard_id) do
+ [] ->
+ {:reply, {:error, :not_found}, state}
+ [{^shard_id, manifest, {:cached, _}, _}] ->
+ pid = case :ets.lookup(:shard_procs, {shard_id, nil}) do
+ [] ->
+ {:ok, pid} = apply(Shard.Manifest.module(manifest), :start_link, [manifest])
+ pid
+ [{{^shard_id, nil}, pid}] ->
+ :ets.delete(:shard_procs, {shard_id, nil})
+ pid
+ end
+ GenServer.call(pid, :delete_shard)
+ :dets.delete(@shard_db, shard_id)
+ {:reply, :ok, state}
+ [{^shard_id, _, _, _}] ->
+ {:reply, {:error, :pinned}, state}
+ end
+ end
+
def handle_cast({:dispatch_to, shard_id, path, pid}, state) do
:ets.insert(:shard_procs, { {shard_id, path}, pid })
state = Map.put(state, pid, {shard_id, path})
@@ -105,6 +126,7 @@ defmodule Shard.Manager do
case :dets.lookup(@shard_db, shard_id) do
[{^shard_id, manifest, why_have_it, _old_state}] ->
:dets.insert(@shard_db, {shard_id, manifest, why_have_it, shst})
+ :dets.sync(@shard_db)
end
{:noreply, state}
end
@@ -169,7 +191,7 @@ defmodule Shard.Manager do
for [id] <- shards do
case :ets.lookup(:shard_procs, {id, nil}) do
[{{^id, nil}, pid}] ->
- GenServer.cast(pid, :delete_shard)
+ GenServer.call(pid, :delete_shard)
_ -> nil
end
:dets.delete(@shard_db, id)
@@ -366,6 +388,13 @@ defmodule Shard.Manager do
end
@doc"""
+ Delete a shard
+ """
+ def delete(shard_id) do
+ GenServer.call(__MODULE__, {:delete, shard_id})
+ end
+
+ @doc"""
Return the list of all shards. Returns a list of tuples:
{id, manifest, why_have_it}
diff --git a/shardweb/lib/controllers/directory_controller.ex b/shardweb/lib/controllers/directory_controller.ex
index d1b7136..ecb9c97 100644
--- a/shardweb/lib/controllers/directory_controller.ex
+++ b/shardweb/lib/controllers/directory_controller.ex
@@ -9,7 +9,7 @@ defmodule ShardWeb.DirectoryController do
view(conn, false, args)
end
- def view(conn, public, %{"owner" => owner, "name" => name}=args) do
+ def view(conn, public, %{"owner" => owner, "name" => name}) do
owner = Base.decode16! owner
manifest = %SApp.Directory.Manifest{public: public, owner: owner, name: name}
shard = SData.term_hash manifest
@@ -28,7 +28,6 @@ defmodule ShardWeb.DirectoryController do
dir_public = (dir_public == "true")
manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name}
- shard = SData.term_hash manifest
pid = Shard.Manager.find_or_start manifest
item_manifest = ShardURI.to_manifest(add_uri)
@@ -41,7 +40,6 @@ defmodule ShardWeb.DirectoryController do
dir_public = (dir_public == "true")
manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name}
- shard = SData.term_hash manifest
pid = Shard.Manager.find_or_start manifest
SApp.Directory.rm_item(pid, rm_name)
@@ -54,7 +52,6 @@ defmodule ShardWeb.DirectoryController do
item_stored = (item_stored == "true")
manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name}
- shard = SData.term_hash manifest
pid = Shard.Manager.find_or_start manifest
SApp.Directory.set_stored(pid, item_name, item_stored)
@@ -68,7 +65,6 @@ defmodule ShardWeb.DirectoryController do
dir_public = (dir_public == "true")
manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name}
- shard = SData.term_hash manifest
pid = Shard.Manager.find_or_start manifest
{:ok, file_manifest, _file_pid} = SApp.File.create(path, mime_type)
diff --git a/shardweb/lib/controllers/page_controller.ex b/shardweb/lib/controllers/page_controller.ex
index 5c90416..baf72b9 100644
--- a/shardweb/lib/controllers/page_controller.ex
+++ b/shardweb/lib/controllers/page_controller.ex
@@ -9,6 +9,14 @@ defmodule ShardWeb.PageController do
render conn, "shard_list.html"
end
+ def shard_action(conn, %{"shard_id" => shard_id, "action" => action}) do
+ shard_id = Base.decode16! shard_id
+ case action do
+ "delete" -> Shard.Manager.delete(shard_id)
+ end
+ redirect conn, to: page_path(conn, :shard_list)
+ end
+
def add_peer(conn, _params) do
try do
ip = conn.params["ip"]
diff --git a/shardweb/lib/router.ex b/shardweb/lib/router.ex
index a6a61f9..0b1f686 100644
--- a/shardweb/lib/router.ex
+++ b/shardweb/lib/router.ex
@@ -19,6 +19,7 @@ defmodule ShardWeb.Router do
pipe_through :browser # Use the default browser stack
get "/", PageController, :shard_list
+ post "/", PageController, :shard_action
get "/peers", PageController, :peer_list
post "/peer/add", PageController, :add_peer
diff --git a/shardweb/lib/templates/page/shard_list.html.eex b/shardweb/lib/templates/page/shard_list.html.eex
index 3fffef7..79fd794 100644
--- a/shardweb/lib/templates/page/shard_list.html.eex
+++ b/shardweb/lib/templates/page/shard_list.html.eex
@@ -23,9 +23,10 @@
<tr>
<th></th>
<th>Shard</th>
+ <th></th>
<th>URI</th>
</tr>
- <%= for {_id, manifest, why_have_it} <- shard_list() do %>
+ <%= for {id, manifest, why_have_it} <- shard_list() do %>
<tr>
<td>
<%= case why_have_it do %>
@@ -37,6 +38,18 @@
<td>
<%= render "shard_entry.html", conn: @conn, manifest: manifest, pk: @pk %>
</td>
+ <td>
+ <%= case why_have_it do %>
+ <% {:cached, _} -> %>
+ <%= form_for @conn, page_path(@conn, :shard_action), [class: "form-inline", style: "display: inline", onsubmit: "return confirm('Delete this shard and all its data?');"], fn f -> %>
+ <%= hidden_input f, :shard_id, value: id |> Base.encode16 %>
+ <%= hidden_input f, :action, value: "delete" %>
+ <%= submit "Delete", [class: "btn btn-xs btn-danger"] %>
+ <% end %>
+ <% _ -> %>&nbsp;
+ <% end %>
+
+ </td>
<td><small class="shard_uri"><%= manifest |> ShardURI.from_manifest %></small></td>
</tr>
<% end %>